We Deployed OpenClaw and Discovered the Hard Part Isn't the Model

AI Collab Score: 8 / 2

​For the last several years, much of the enterprise AI conversation has centered on models and GPUs.

Which model should we use?

How many GPUs do we need?

How quickly can we generate tokens?

Those are important questions, and they have been central to much of my lab work: local models, inference performance, prefill and decode behavior, KV cache utilization, and the infrastructure required to turn compute into usable AI capacity.

But as AI moves from answering questions to performing work, the architecture changes.

The model still matters.

But the model is no longer the whole system.

An enterprise agent may need to interpret a request, select an appropriate model, determine whether external verification is required, invoke a tool, evaluate the result, and either complete an action or decide what should happen next.

That is no longer just inference.

It is orchestration.

Recently, we began deploying OpenClaw in the lab to explore that orchestration layer: agents, tools, model routing, external services, policy boundaries, and the operational controls required to make all of those components work together.

What began as an OpenClaw deployment quickly became an experiment in AI orchestration.

It became an experiment in the systems required to make AI work beyond the model.

And we learned quite a bit along the way.

---

Why OpenClaw?

​Most of my previous lab work has focused on inference.

The basic interaction is relatively straightforward:

Even though there is significant infrastructure underneath that interaction, the application path itself is fairly easy to understand.

Agents introduce another layer.

That loop may repeat several times before the task is complete.

That changes how we design, operate, secure, and measure the infrastructure.

It also changes what we need to measure.

OpenClaw gave us a self-hosted environment where we could start experimenting with those questions instead of discussing agents purely at the conceptual level.

---

​What We Actually Deployed

The first step was bringing up the OpenClaw runtime and Gateway.

But simply getting OpenClaw running wasn't the goal.

We wanted to see what happened when we connected an agent environment to several different sources of intelligence and began extending the platform ourselves.

Our lab environment eventually included access to:

• a local LLM
• ChatGPT
• Gemini
• Perplexity

We also built and deployed our own custom capability around deterministic routing and verified external research.

The architecture started looking less like a chatbot and more like this:

In this context, validation can include policy checks, response-schema checks, source and URL evaluation, dependency-status checks, and confirmation that a returned result is usable for the requested workflow.

That distinction matters because having the model simply inspect tool output is not the same thing as having a controlled validation path.

We weren't just testing whether a model could answer a question anymore.

We were testing how several components could work together to complete a task.

---

​The Lab Environment

​The environment was intentionally small enough to experiment with but complex enough to expose some of the same problems an enterprise deployment would eventually need to address.

OpenClaw was self-hosted in the lab and connected to both a local inference path and selected external model and research providers.

That allowed us to experiment with two very different execution models.

Some requests could remain on an approved local path.

Others could use external capabilities when additional reasoning, research, or current information was needed.

We began building controls around how—and when—those capabilities were exposed rather than simply giving the agent unrestricted access to everything available.

That became increasingly important as we moved from asking:

• Can the agent use this?​

to:

• When should the agent be allowed to use this?

Those are very different questions.

---

One Agent, Multiple Brains

Connecting a local model, ChatGPT, Gemini, and Perplexity immediately changed the architecture.

If an application only has access to one model, model selection is easy.

There is nothing to select.

Once several capabilities exist, a new question appears:

• Which capability should handle which task?
  

​
I don't think the answer should necessarily be tied permanently to a particular vendor.

It is more useful to think in terms of capability classes.

The following is a potential capability-routing pattern, not a universal assignment of workloads to providers.

Workload requirement	Potential capability
Sensitive or private information	Local or approved private inference path
Complex reasoning	Approved frontier reasoning model
Current external information	Research or retrieval capability
Repetitive, lower-complexity task	Smaller or lower-cost model
Multimodal workload	Multimodal-capable model
External verification required	Verified research path

In our lab, ChatGPT, Gemini, Perplexity, and the local LLM provided different paths through that architecture.

But the larger lesson wasn't about any individual provider.

It was this:

• Once multiple models and tools are available, model selection stops being merely a configuration decision. It starts becoming an architectural and policy decision.
  

That led directly into our next experiment.

---

Building the Deterministic Router

One of the things I didn't want was for every routing decision to depend entirely on probabilistic model behavior.

There are places where probabilistic reasoning is useful.

There are also places where I want the system to follow a rule.

That led us to build a custom Deterministic Router.

The router went through several iterations.

An earlier implementation primarily used hooks. Later versions introduced a callable capability we called:

• verified_research

This was our custom tool, not a native OpenClaw feature.

The idea was to create a controlled research path that could incorporate things such as:

• external research
• secure retrieval
• URL policy
• source evaluation
• source ranking
• research orchestration
• deterministic routing behavior

The deterministic layer was not intended to replace model judgment.

It was intended to bound it.

The agent could reason within the capabilities available to it, while policy could determine when external retrieval, data egress, particular tools, or additional controls were required.

Conceptually, that begins to look like:

The following is an illustrative policy model, not a claim that every rule is implemented in the current lab environment.

Condition	Deterministic action	Reason
Request contains restricted data	Use an approved private or local path	Prevent inappropriate external disclosure
Current facts require verification	Invoke verified_research	Require retrieval and evidence
Low-risk repetitive workload	Use a lower-cost approved model	Control cost and latency
Requested tool is prohibited	Deny the request or require approval	Enforce least privilege
External research is unavailable	Return a bounded failure	Avoid claiming verification occurred

We aren't claiming that every one of those policies is implemented today.
​
But that is the architectural direction the experiment started exposing.

---

Then Something Broke

One of the most useful lessons came from something that initially looked like a successful deployment.

We had built the plugin.

The package installed.

OpenClaw recognized it.

The runtime showed that the plugin had loaded.

Everything looked healthy.

Except there was a problem.

The agent didn't actually have the verified_research capability we expected.

The plugin was present.

The runtime knew about it.

But the tool was not being exposed correctly to the agent.

In practice, this reinforced that a plugin's runtime behavior and the metadata required for agent-facing tool discovery must both be correct; a loaded extension is not automatically an available agent capability.

From a traditional infrastructure perspective, you could have looked at the environment and concluded:

Healthy.

From the perspective of the workload, however, a critical capability was missing.
​
That failure changed how I started thinking about agent health.

---

Lesson 1: Installed Does Not Mean Working

Traditional infrastructure gives us several familiar states.

A package exists.

A process starts.

A service responds.

A health check turns green.

For an agent platform, those signals are necessary.

But they aren't sufficient.

The progression we experienced looked more like this:

​Each state means something different.

State	Meaning	Example validation
Installed	Package exists	Plugin inventory confirms presence
Enabled	Configuration permits use	Plugin is enabled in configuration
Loaded	Runtime initializes it	Runtime reports the plugin loaded
Callable	Agent receives the tool definition	Tool appears in the agent’s available capability set
Executable	Invocation reaches the handler	Controlled invocation returns a valid response
Functional	Capability completes its intended workflow	Research produces a usable validated result
Reliable	Capability consistently meets defined operational expectations	Repeated testing under normal and failure conditions

That last state is important.

A capability working once does not necessarily make it reliable.
​
And that leads to something I think enterprise operations teams will eventually need to pay much more attention to:

Capability Health

A normal health check might tell me:

OpenClaw Gateway: UP

But what I actually want to know is:

Can the agent authenticate?

Can it discover the required tool?

Can it invoke the tool?

Can the tool reach its dependency?

Did the dependency return a valid response?

Can the system evaluate that response?

Can it recover if something fails?

Can it complete the business task?

Those are much more meaningful questions.
​
A green process indicator tells us very little about whether the agent can actually work.

---

Lesson 2: The Agent Platform Is an Integration Platform

​This became increasingly obvious as the experiment progressed.

At first, it is easy to picture:

But a useful enterprise agent may eventually need to interact with:

• identity systems
• secrets managers
• internal APIs
• databases
• knowledge repositories
• vector stores
• SaaS platforms
• ticketing systems
• observability platforms
• approval workflows
• security controls
• multiple model providers

Suddenly, the LLM doesn't look like the application anymore.

It looks like one component in a distributed system.

That has major implications for enterprise architecture.

• Identity matters.
• Networking matters.
• API reliability matters.
• Authorization matters.
• Observability matters.
• Data locality matters.
• Failure handling matters.
• The GPU still matters too.

But once the model begins interacting with other systems, the GPU is only one component in a much larger execution path.

---

Lesson 3: Action Requires Deterministic Guardrails

Generative AI is inherently probabilistic.

For a conversational system, variability is often acceptable.

Ask the same question twice and you may receive two slightly different answers.

That usually isn't a problem.

But compare these two requests:

“Explain how to restart this server.”

and

“Restart this server.”

The first produces information.

The second produces an outcome.

That distinction changes the risk model.

As AI gains the ability to perform actions, enterprises will need much stronger controls around:

• which tools an agent can use
• when those tools may execute
• which systems can be accessed
• which data can leave a boundary
• what sources are trusted
• what actions require human approval
• what happens when a dependency fails
• how actions are recorded
• how actions can be reviewed

A useful starting point is to separate read-only retrieval from state-changing operations.

An agent may be allowed to search, summarize, analyze, or prepare a recommendation automatically, while actions that alter systems, permissions, records, infrastructure, or external communications require an explicit approval boundary.

This is where deterministic controls become especially interesting.

Reasoning can remain probabilistic.
​
But certain boundaries do not have to be.

---

Lesson 4: Multi-Model AI Needs a Control Plane

Our multi-model experiment created another realization.

Enterprises probably aren't going to operate with one AI model.

They may operate with dozens.

Some may be hosted internally.

Others may come from external providers.

Some may specialize in code.

Some may specialize in reasoning.

Some may be multimodal.

Some may be inexpensive enough for high-volume tasks.

Others may be reserved for workloads where additional capability justifies the cost.

Now add constraints such as:

• privacy
• security
• latency
• cost
• model availability
• data sovereignty
• context requirements
• tool support
• business policy

Someone—or something—has to decide where a workload goes.

That is why I think one of the important architectural components emerging around enterprise
AI will be an AI control plane.

The question stops being:

• Which model is best?

and becomes:

• Which approved capability is appropriate for this workload, under these conditions, right now?

That's a substantially different infrastructure problem.

---

Lesson 5: We Need to Benchmark Work, Not Just Models

My previous lab testing has focused heavily on traditional inference metrics:

• time to first token
• prefill
• decode performance
• tokens per second
• concurrency
• GPU utilization

Those metrics remain important.

But an agent can generate tokens extremely quickly and still fail at the task.

For an agentic system, the more important measurement may eventually become:

• Did the work get completed correctly?

That requires us to look at the entire execution chain.

Now our measurements begin to change.
​
Instead of only asking about tokens per second, I want to start looking at:

• time to first action
• total task completion time
• number of model calls
• tokens consumed per completed task
• tool calls per task
• tool latency
• retries
• failed calls
• sources evaluated
• sources rejected
• routing decisions
• model selection
• CPU utilization
• GPU utilization
• memory utilization
• network traffic

But those infrastructure and execution metrics still don't tell the whole story.

A mature scorecard should measure not only whether a task completed, but whether it completed correctly, within policy, with appropriate evidence, at an acceptable cost, and with a safe failure mode when completion was not possible.

That means adding outcome-oriented measures such as:

• task success rate
• policy-compliant task success rate
• verified-answer rate for retrieval-required tasks
• unsafe-action prevention rate
• false escalation rate
• failed-safe rate
• mean time to recovery after dependency failure
• cost per successfully completed task
• human-intervention rate
• audit completeness rate

Those metrics move us away from simply asking how fast the model generated tokens.

​They start measuring whether the system produced useful, trustworthy work.

---

What Comes Next

The next phase of the lab isn't simply installing another model.

It is giving the system measurable work.

For example:

• Research a technical question, determine whether external verification is required, select an appropriate capability, retrieve supporting information, evaluate the sources, reject information that fails policy, and return a verified result.

Then we instrument the entire execution path.

After establishing a baseline, we can deliberately introduce failure.

What happens if the local model is unavailable?

What happens if an external model provider becomes unreachable?

What happens if Perplexity doesn't respond?

What happens when a tool returns malformed information?

What happens when the research path is blocked?

What happens when two capabilities could handle the same request?

What happens when multiple agents execute simultaneously?

And perhaps most importantly:

Does the system fail safely?
​
Those experiments will tell us far more about enterprise agent infrastructure than another isolated tokens-per-second benchmark.

---

From Benchmarking Models to Benchmarking Work

I would not call what we have today a production enterprise agent architecture.

And I think that distinction is important.

What we have done is successfully deploy the OpenClaw platform, connect multiple local and external intelligence paths, extend the environment with our own routing and research capability, work through failures in that integration, and begin exploring what it takes to operate an agent system rather than simply host an LLM.

The next phase is about reliability.

We want to know not only whether a model can generate a good answer, but whether the entire system can:

• select the right capability
• enforce the right policy
• invoke the right tool
• validate the result
• survive dependency failures
• recover appropriately
• complete useful work consistently

For years, AI infrastructure teams have benchmarked how fast AI can think.

Agentic AI adds a harder question:

How reliably can AI work?

I suspect that will become one of the defining benchmarks of enterprise AI.

We spent the last few years benchmarking how fast AI can think.
​
Now it's time to start measuring how reliably AI can work.