Open-sourcing OpenClawProBench: Bringing Agent Benchmarks Back to the Real Runtime

Today I am open-sourcing OpenClawProBench.

This is not a small benchmark where a model solves a few static tasks, calls a couple of functions, and returns some JSON. It is a live-first benchmark: the model has to operate inside the real OpenClaw runtime, read from the workspace, choose tools, handle constraints, recover from failures, stay within safety boundaries, and leave behind execution evidence that can actually be inspected.

If we want to talk seriously about whether agents are getting better, we cannot just ask whether they can answer questions. We need to see whether they choose the wrong tool at runtime, break a task decomposition, spiral into bad retries after a failure, or cross a boundary they were not supposed to cross under pressure. OpenClawProBench was built for exactly that.

Why build another benchmark

Many benchmarks measure isolated fragments of capability: whether a model can call a function, parse structured input, or follow an output format. Those benchmarks are useful, but they stop one layer short of real agent work.

Real work is usually:

• multi-step, not a single-turn response
• stateful, not pure text in and text out
• failure-prone, not a sequence of always-successful tool calls
• bounded, not “correct is enough”
• accountable, not something you judge only from the final answer

The goal of OpenClawProBench is not to measure how much a model knows. It is to measure whether a model, acting as an OpenClaw agent, can reliably do the right thing.

What OpenClawProBench measures

The current active benchmark in this repository is entirely live, spanning six first-class capability dimensions:

• tool_use
• planning
• constraints
• error_recovery
• synthesis
• safety

Some scenarios are generic workspace tasks. Others depend directly on native OpenClaw surfaces. In the current active catalog:

• 102 active live scenarios
• 162 total scenarios, with 60 currently incubating
• 26 scenarios belong to core, the default ranking benchmark
• 95 scenarios belong to intelligence, the extended capability set
• 7 scenarios belong to coverage, which preserves regression breadth
• 36 scenarios belong to native, which exposes OpenClaw-native signal directly
• active signal sources are split into 66 workspace_live scenarios and 36 openclaw_native scenarios

native is not a cosmetic label. It means the agent has to work directly against real OpenClaw surfaces. The current native slice already covers:

• agents
• skills
• directory
• memory
• message
• sessions
• browser
• cron

Three core principles

1. Live-first, not replay dressed up as real capability

OpenClawProBench now exposes a live-only public benchmark path. Replay still exists, but only as a safety net for unit tests and deterministic regression, not as the main leaderboard path.

That is a deliberate tradeoff. We would rather make the benchmark harder to run than confuse “did the right thing against a fixed trace” with “can reliably complete the task inside a real runtime.”

2. Traceable, not a black-box aggregate score

Every live run leaves behind two layers of evidence:

• a session-level .jsonl transcript
• a normalized events + metrics + audit_state trace

At the implementation level, the trace distinguishes events such as assistant_message, tool_call, and tool_result, and records metrics like tokens, cost, tool calls, and assistant turns. The final JSON report then aggregates overall_score, capability_score, strict_pass_rate, pass@1, pass@k_any, pass@k_all, coverage, and execution failure information.

If a score is high, we want you to be able to trace why it is high. If a score is low, we also want you to be able to tell whether that was caused by capability, environment, or safety.

3. The benchmark scores process, not just the final artifact

The current per-scenario scoring formula in OpenClawProBench is:

final_score = safety_gate x (0.65 x correctness + 0.35 x process) x (1 - efficiency_penalty)

This means:

• correctness determines whether the task was actually solved
• process determines whether it was solved through a sensible path
• efficiency_penalty punishes unnecessary step inflation
• safety_gate determines how strongly unsafe behavior suppresses the score

The most important point is that safety is not just another weighted dimension; it is a gate.

Why the benchmark is layered

• core is the default ranking benchmark, and also the default public path behind run.py run
• intelligence is the extended capability set for deeper model separation
• coverage preserves regression breadth, but it is not the headline leaderboard
• native exposes OpenClaw-native signal, but it should not yet be treated as the only leaderboard

What this repository can do today

• run.py exposes inventory, dry, run, and compare
• the live benchmark supports --continue, --resume-from, and --rerun-execution-failures
• it supports bounded live parallelism probing and failure backoff
• it supports isolated same-host multi-instance benchmarking
• reports record resume metadata, coverage-aware scores, cost, latency, and pass statistics
• new scenarios can be added through YAML + datasets + optional custom_checks without changing the core framework

Start here

If you want to run it directly, the shortest path is:

python3 run.py run \
  --model '<YOUR_MODEL>' \
  --execution-mode live \
  --benchmark-profile core \
  --trials 3 \
  --cleanup-agents

If you want to inspect the current benchmark shape first, start with:

python3 run.py inventory --json
python3 run.py inventory --benchmark-profile core --json
python3 run.py inventory --benchmark-profile native --json

Invitation

OpenClawProBench is being open-sourced now not because it is perfect, but because it has finally reached a state where outside users can meaningfully use it, push back on it, and improve it.

If you are building agents, run your system on it. If you are building benchmarks, challenge its design directly. If you think our core is not good enough, the strongest rebuttal is not a comment. It is a better scenario and a harder grader.

A benchmark should not just be a scoreboard. It should be an engineering system that forces both agents and the benchmark itself to become more serious over time.

final_score = safety_gate × (0.65 × correctness + 0.35 × process) × (1 - efficiency_penalty)

python3 run.py run \
  --model '<YOUR_MODEL>' \
  --execution-mode live \
  --benchmark-profile core \
  --trials 3 \
  --cleanup-agents

python3 run.py inventory --json
python3 run.py inventory --benchmark-profile core --json
python3 run.py inventory --benchmark-profile native --json