ClawProBench Closed Dataset Release and Analysis

ClawProBench now includes a closed dataset leaderboard. It is not a replacement for the open leaderboard. It adds a more production-oriented view of model capability: when a model works through customer support, vendor security questionnaires, log redaction, approval boundaries, incident recovery, billing disputes, CRM updates, privacy exports, and other enterprise workflows, can it make stable, compliant, executable decisions under limited information and high-risk constraints?

In short, the open dataset asks whether a model is a strong general OpenClaw agent. The closed dataset asks whether it is ready for production workflows.

What Was Released

This closed release contains 68 tasks and 33 model results. The open leaderboard currently contains 102 tasks and 66 model results.

Compared with the open dataset, the closed dataset changes the evaluation pressure in two important ways.

First, Safety becomes much more central. The open dataset has 12 safety tasks, while the closed dataset has 17. More importantly, these tasks are not just about refusing obviously dangerous requests. They test whether a model can still be useful while keeping secrets, logs, customer data, vendor material, and internal sources inside the right boundary.

Second, the closed dataset is almost entirely hard tasks. The open dataset has 52 expert tasks, 39 hard tasks, 8 medium tasks, and 3 easy tasks. The closed dataset has 64 hard tasks and 4 expert tasks. In this context, hard does not simply mean longer prompts. It means the model must choose between competing real-world constraints: answer without leaking, make progress without overstepping authority, synthesize evidence without pretending conflicts are resolved, and use tools without expanding data access.

What the Closed Dataset Measures Differently

The open dataset evaluates broad OpenClaw agent ability: planning, tool use, error recovery, constraint following, synthesis, and safety. It is best for answering one question: is this model strong as a general work agent?

The closed dataset goes further. It focuses on whether the model is reliable under enterprise risk. It answers a different question: can this model handle real customers, real permissions, real data boundaries, and real incidents?

Those two questions often have different answers.

Many closed tasks are not difficult because the model lacks knowledge. They are difficult because the model needs to know what not to do:

• When a customer asks for a debug bundle, provide enough reproduction context without exposing secrets, internal logs, or customer data.
• When answering a vendor security questionnaire, separate disclosable facts from internal-only details and approval-required items.
• When a ticket or attachment contains prompt injection, treat that content as untrusted and do not execute it as an instruction.
• When billing, refund, SLA, contract, and support records conflict, identify confirmed facts and unresolved conflicts instead of inventing one clean conclusion.
• In CRM, account merge, and privacy request workflows, use the smallest necessary read or write scope instead of acting broadly just because a tool exists.

The closed dataset penalizes two common model behaviors. One is excessive compliance: the user asks for something, and the model gives too much. The other is excessive action: the model sees tools and immediately updates, merges, exports, or writes. The closed dataset rewards models that are restrained but still useful.

Closed Final Score

The closed leaderboard keeps three core signals:

• AvgScore: average task quality across rubric items.
• Pass^3: stable three-run success, which reflects consistency.
• Pass@3: at least one successful path across three runs, which reflects upside and sampling potential.

The open leaderboard puts more weight on stable three-run success. The closed leaderboard increases the weight of Pass@3 while still retaining Pass^3. The reason is practical: the closed set is smaller, more concentrated, and more realistic. If a model can find a compliant path in at least one of three attempts, that is useful signal. But if Pass^3 is low, the model is still not stable enough for unsupervised production use.

The current page formulas are:

Open Final Score = 100 x AvgScore^0.40 x ((Pass^3)^(1/3))^0.45 x (1 - (1 - Pass@3)^(1/3))^0.15
Closed Final Score = 100 x AvgScore^0.40 x ((Pass^3)^(1/3))^0.25 x (1 - (1 - Pass@3)^(1/3))^0.35

So the closed ranking is not just a single-run score, and it is not only a stability ranking. It balances average quality, consistency, and three-try recoverability.

Closed Leaderboard Results

The current closed Top 12 are:

The first-place model, doubao-seed-2.0-code, has a clear shape. It is not carried by one isolated metric. It is especially strong where the closed dataset matters most: Safety is 79.3, Synthesis is 74.9, and Pass@3 is 81.8%. That means it is better at keeping customer data, vendor boundaries, conflicting evidence, and external-facing language under control while still making progress.

kat-coder-pro-v2 is one of the most interesting results. It is not a top open-leaderboard model, but it rises to second in the closed leaderboard. The reason is not Safety, where it scores 52.7. The lift comes from Planning at 76.4, Tool Use at 76.8, Constraints at 73.5, and Synthesis at 70.4. Its ability shape matches closed enterprise tasks very well, although safety remains its main weakness.

gpt-5.3-codex-xhigh is the more balanced profile. It does not dominate one single metric, but Safety is 68.1, Tool Use is 75.0, Constraints is 68.8, Error Recovery is 75.1, and Synthesis is 70.5. It looks more like a stable enterprise agent. Its Pass^3 is not the highest, but AvgScore and Pass@3 are both strong.

deepseek-v4-pro reaches 79.0 Pass@3, so its ceiling is high. But its Pass^3 is 34.1, Safety is 48.6, and Constraints is 55.2. It often finds a good path, but not reliably enough, and it is more likely to slip on safety and policy boundaries.

gpt-5.5-xhigh is strong in Tool Use, Error Recovery, and Synthesis, at 77.7, 75.0, and 74.1. Its overall rank is pulled down by Safety and Constraints. This kind of model can be useful for execution-heavy and recovery-heavy workflows, but workflows involving privacy, logs, secrets, and disclosure need stronger outer controls.

Why Open Scores Do Not Fully Transfer

The most important finding is that open ranking strength does not automatically transfer to the closed ranking.

Across the 33 paired models that have both open and closed results, the average closed-minus-open dimension movement is:

This is the key result. The closed dataset is not simply lower across the board. Planning, Tool Use, and Synthesis are actually higher on average. Many models can adapt to realistic task decomposition, tool choice, and evidence consolidation. The main break is Safety, with an average drop of 9.8 points. That is the largest gap by far.

This explains why some models that rank very high on the open leaderboard fall much lower on the closed leaderboard.

To avoid turning the open leaderboard front-runners into the focus of this release note, the table below uses non-front-runner examples and the aggregate pattern:

These models are capable general agents. The issue is that closed tasks often require the model to do less, but do it correctly. Do not reveal internal logs. Do not turn a customer request into a broad privacy export. Do not promise an exception without approval. Do not execute instructions embedded in an untrusted attachment. Models that are too eager to complete the task can lose points on safety and constraints.

Pass^3 shows the same issue. For several high open-ranking models, closed Pass^3 falls sharply. That means the problem is not just one bad answer. It is stability under enterprise boundaries. A model can occasionally answer correctly, but three stable boundary-preserving runs are much harder.

Why Some Models Improve on the Closed Dataset

Some models move up sharply on the closed leaderboard.

The common factor is not simply higher intelligence. It is a capability shape that fits the closed dataset.

kat-coder-pro-v2 gains mainly from tool action, constraint following, and evidence synthesis. Closed tasks often ask whether the model should read minimal evidence, whether it has authority to continue, or whether conflicting facts can be stated as resolved. A model that handles those decisions well can rise even if it is not near the top of the broader open leaderboard.

doubao-seed-2.0-code improves in a more direct way: Safety and Synthesis both rise. It can complete the task while staying inside enterprise boundaries. That is exactly what the closed dataset rewards.

MiniMax-M2.1 and MiniMax-M2.5 show another useful pattern. A model can be less dominant on broad public tasks but still be competitive in structured enterprise workflows, tool choice, and evidence merging. For real deployment, that can matter more than a single general ranking.

A Production-Readiness Benchmark

If we only look at the open leaderboard, model ability can look like planning, tool use, and task completion. In production, the model must also:

1. avoid leaking content that should not be exposed;
2. avoid executing actions it has no authority to perform;
3. treat untrusted input as untrusted input;
4. avoid making company commitments without approval;
5. avoid turning conflicting evidence into a fake clean conclusion;
6. avoid expanding data access just to look helpful.

The closed dataset is built around these pressure points. It is especially useful for selecting models that can enter enterprise workflows, not just models that score well on public tasks.

The current results suggest three requirements for closed-set performance.

First, the model needs safety restraint. Safety is the biggest differentiator. Many models do not fail because they cannot solve the task. They fail because they choose the wrong boundary between helping and leaking.

Second, the model needs evidence discipline. A high Synthesis score is not just fluent writing. It means the model can separate source of truth, conflicting records, approval state, and uncertainty.

Third, the model needs tool minimalism. A high Tool Use score is not about using more tools. It is about knowing when to read only, when to preflight, when to escalate, and when not to write.

How to Read the Two Leaderboards

If you are choosing a general OpenClaw agent, the open leaderboard remains important. It covers 102 tasks and more models, so it reflects broader capability.

If you are choosing a model for internal enterprise workflows, customer support, safety and compliance, operations support, vendor questionnaires, incident reviews, or CRM and ticket processing, the closed leaderboard should be read first.

A practical reading:

• Customer data, logs, secrets, and vendor material: prioritize Safety and Constraints.
• Tickets, CRM, account merges, and evidence packets: prioritize Tool Use and Constraints.
• Incident reviews, billing disputes, and contract conflicts: prioritize Synthesis and Error Recovery.
• On-call, scheduling, resource allocation, and approval windows: prioritize Planning.
• Unsupervised deployment: look closely at Pass^3.
• Reviewed or multi-sample workflows: Pass@3 is also important.

Final Score is useful for quick sorting, but deployment decisions should look at the model shape. A high closed score does not mean a model is strong in every enterprise scenario. A high open score does not mean it can naturally handle privacy, authority, and compliance boundaries.

Conclusion

With the closed dataset, ClawProBench no longer only asks which model is strongest overall. It also asks which model is more ready for real enterprise agent work.

In this release, doubao-seed-2.0-code ranks first because Safety, Synthesis, and Pass@3 are all strong. kat-coder-pro-v2 rises sharply because its Planning, Tool Use, Constraints, and Synthesis match the closed enterprise workload. gpt-5.3-codex-xhigh shows a more balanced enterprise-agent profile. At the same time, some models that are very strong on the open leaderboard drop on the closed dataset, mainly because safety boundaries, constraint following, and stable behavior do not fully transfer.

That is the point of the closed dataset. It does not let a model only show strength on public tasks. It puts the model closer to production pressure and checks whether it can both act and keep boundaries.

For real deployment, the open and closed leaderboards should be read together. The open leaderboard shows broad capability. The closed leaderboard shows production readiness. The bigger the gap between the two, the less we should trust any single aggregate score as a complete description of a model.