Benchmark the agent, not just the model.
Orivoxx runs an agent against a versioned specification in a hardened sandbox, scores the produced repository deterministically, and attributes every failure to a layer and a mode — with a replay that proves the fix.
latest attributed run
Organizer-runitsdangerous-future-signature
python (chaos: failing_tools)
anthropic/claude-sonnet-4.5
0.0
harness × recovery
apply_patch failed (EIO: intercepted tool failure (injected)) and was not recovered
proof: 0.0 → 100.0 after retry at step 2
\ THE PROBLEM \ \ 01 \
A trace tells you what happened. It does not tell you why.
Developers stare at a 47-step trace and guess which layer failed. The model? The harness? A tool? The specification itself? Every guess costs another full run.
the hand-offs where signal is lost
The agent sees an error, retries the wrong thing, or silently drops the result. The artifact fails, but the cause is not in the diff.
Model blamed for harness bugs
A swallowed tool error looks like bad reasoning when the harness never retried.
Tool failures look like decisions
A transient EACCES and a wrong design choice produce the same failed artifact.
No way to test a fix
Re-running the whole agent to test one policy change is slow, noisy, and expensive.
\ THE SOLUTION \ \ 02 \
A deterministic benchmark with an attribution layer.
The score is the substrate. The attribution is the product. Every failure gets a cell in a 5×4 taxonomy and a counterfactual replay that crosses the pass threshold.
Deterministic scoring
Declared checks and static analysis. No LLM in the loop; every point traces to a rule.
SCORE
Counterfactual attribution
Layer × mode with evidence, then a replay that changes one thing and proves the delta.
ATTRIBUTE
Reproducibility
Versioned tasks, rulesets, adapters, and a repository fingerprint on every row.
REPLAY
attribution card
evidence step 2, write_file → EACCES, no retry
proof replay with retry policy → PASS
delta 0.0 → 100.0
scope intervention at 2, replay 3–7, static scoring
\ HOW IT WORKS \ \ 03 \
Five steps. One reproducible record.
01
Spec
A versioned task pins the spec, ruleset, scoring config, and starting files.
02
Run
The agent executes in Docker; model calls pass through one allowlisted proxy.
03
Score
Static checks and structural analysis produce the artifact score.
04
Attribute
Rules map the failure to layer × mode with step-level evidence.
05
Replay
A recorded intervention re-runs the tail and proves pass/fail.
\ MEASUREMENT \ \ 04 \
Six metrics. Four chaos modes. One reliability profile.
The track asks for six metrics and resilience testing. Those are the substrate. Attribution is the layer above them: it names where the loss entered and proves a fix without guessing.
Artifact score
0–100, severity weighted
Steps
model + tool steps
Tool calls
count + error rate
Tokens
prompt + completion
Cost
provider usage
Duration
wall clock
tool_error
One write fails with EACCES; the harness either retries or loses the file.
failing_tools
The real-task apply_patch is intercepted; the replay retries it.
latency
Every model call gets a fixed delay; the stopping policy is tested.
context_pressure
Context is truncated and the step budget shrinks; completeness is tested.
\ REAL REPOSITORY TASKS \
A base commit, an issue, a real test.
is-email-predicate
is: add an email predicate and EmailString type
The guard must validate a practical email shape while the public TypeScript surface gains a branded EmailString type.
itsdangerous-future-signature
itsdangerous: add SignatureNotYetValid
A future-dated timestamp is currently reported as expired; it needs its own exception class, package export, and signer branch.
slugify-max-length
slugify: add a maxLength option
A new option must cap the final slug at a boundary, with the runtime and the published type declaration kept in sync.
\ LEADERBOARD \ \ 05 \
Reliability, not a single lucky run.
| # | Adapter | Task | Model | Score | Pass@k | Reliability |
|---|---|---|---|---|---|---|
| 1 | OpenAI Codex CLI (openai/gpt-5.1-codex) | is-hexadecimal-single-char | openai/gpt-5.1-codex | 100.0 | 100% | 1 runs |
| 2 | Aider 0.86.2 (anthropic/claude-sonnet-4.5) | is-word-character-single-char | anthropic/claude-sonnet-4.5 | 100.0 | 100% | 1 runs |
| 3 | OpenAI Codex CLI (openai/gpt-5.1-codex) | is-number-reject-hex | openai/gpt-5.1-codex | 100.0 | 100% | 1 runs |
| 4 | OpenAI Codex CLI (openai/gpt-5.1-codex) | is-whitespace-single-char | openai/gpt-5.1-codex | 100.0 | 100% | 1 runs |
\ ATTRIBUTION \ \ 06 \
Every failure named. Every fix proven.
An attribution without a counterfactual is an opinion. Orivoxx stores the baseline trace, the intervention, and the replay score. If no intervention crosses the threshold, the result says so.
Open a replay proof →proof
Sufficient0.0 → 100.0
\ FOR DEVELOPERS \ \ 07 \
Run it, attribute it, replay it.
The organizer CLI uses the same deterministic engine as the web app. Real adapters run in Docker; the public app reads the immutable result.
Adapter guide →$ orivoxx bench --task taskboard --chaos tool_error
run 0.0 · harness × recovery · proof 100.0
$ orivoxx replay --result res_64068a
intervention retry_policy@2 · replayed 6 · PASS
$ orivoxx diff --a res_64068a --b res_df4fc5
score +100.0 · real test passes · no model in score
\ FAQ \ \ 09 \
Is this another trace viewer?+
No. A trace viewer shows steps. Orivoxx names the layer and mode, then proves the attribution with a replay.
How is attribution verified?+
The proof is a counterfactual replay that crosses the pass threshold. If no intervention works, the result says unattributed.
Does it work with any framework?+
The adapter contract is framework-agnostic: one argv command inside the sandbox. Python and Aider ship today.
What is deterministic?+
The score, the rules, the replay, and the attribution. No model output enters the score path.
What if the model provider is down?+
Stored results stay available. The score path makes no model call; a missing trace is reported as unavailable.
What can't it attribute?+
Failures outside the trace, unseen conditions, and cells the ruleset does not cover. Recall is published, not assumed.
Benchmark the agent. Prove the failure.
is: add an email predicate and EmailString type is waiting. Run the baseline, inject a fault, and open the replay.
Open source · MIT · Benchmark the agent, not just the model