\ BENCHMARK CONTROL ROOM \
Benchmark the agent. Attribute the failure.
Versioned tasks, sandboxed runs, deterministic scores, and counterfactual proofs. Start with a baseline, inject a fault, then open the replay.
- Tasks
- 8
- Adapters
- 4
- Results
- 21
- Attributed
- 6
- Pass rate
- 71%
- Store
- upstash
\ 01 \ TOP RESULTS
The current board.
| Rank | Adapter | Task | Model | Score | Pass@k | Reliability |
|---|---|---|---|---|---|---|
| 1 | OpenAI Codex CLI (openai/gpt-5.1-codex) | is-hexadecimal-single-char | openai/gpt-5.1-codex | 100.0 | 100% | 1 |
| 2 | Aider 0.86.2 (anthropic/claude-sonnet-4.5) | is-word-character-single-char | anthropic/claude-sonnet-4.5 | 100.0 | 100% | 1 |
| 3 | OpenAI Codex CLI (openai/gpt-5.1-codex) | is-number-reject-hex | openai/gpt-5.1-codex | 100.0 | 100% | 1 |
| 4 | OpenAI Codex CLI (openai/gpt-5.1-codex) | is-whitespace-single-char | openai/gpt-5.1-codex | 100.0 | 100% | 1 |
\ 02 \ TASKS
Versioned, immutable, scored.
is-email-predicate · v1.0.0
intermediateis: add an email predicate and EmailString type
The guard must validate a practical email shape while the public TypeScript surface gains a branded EmailString type.
- requirements
- 1
- pass threshold
- 70
itsdangerous-future-signature · v1.0.0
advanceditsdangerous: add SignatureNotYetValid
A future-dated timestamp is currently reported as expired; it needs its own exception class, package export, and signer branch.
- requirements
- 1
- pass threshold
- 70
slugify-max-length · v1.0.0
intermediateslugify: add a maxLength option
A new option must cap the final slug at a boundary, with the runtime and the published type declaration kept in sync.
- requirements
- 1
- pass threshold
- 70
\ 03 \ RECENT RUNS
The last five hand-offs.
\ 04 \ ATTRIBUTION
One intervention. One proof.
Replay reconstructs the workspace from the recorded files, applies the intercepted action, and re-runs the real test command in the sandbox. Before step k the prefix is byte-identical. No model call enters the score.
Open a real proof →baseline
0.0
→
intervention
retry @ 2
→
replay
3 steps
→
proof
100.0
PASS
\ 05 \ QUICK START
Three commands.
Organizer runs execute in Docker. The public app reads the immutable result and never runs an arbitrary command.
$ orivoxx bench --task itsdangerous-future-signature --chaos failing_tools
0.0 · harness × recovery · 3 replayed steps
$ orivoxx replay --result res_c1ba8d7
retry_policy@2 → 100.0 · PASS
$ orivoxx diff --a res_c1ba8d7 --b res_85394c
+100.0 score · real test passes
\ YOUR TASKS \
Submit a repo issue or review your validations.
\ 06 \ RESILIENCE
Inject the fault. Measure the recovery.
tool_error
A write fails with EACCES. The harness either retries or loses the artifact.
Run with tool_error →failing_tools
The real-task apply_patch is intercepted. The replay retries it and re-runs the real test.
Run with failing_tools →latency
Every model call is delayed. The stopping policy is tested against the clock.
Run with latency →context_pressure
Context is truncated and the step budget shrinks. Completeness is tested.
Run with context_pressure →