Skip to content
HackCulture Global AI Hackathon — Benchmarking track

Benchmark the agent, not just the model.

Orivoxx runs an agent against a versioned specification in a hardened sandbox, scores the produced repository deterministically, and attributes every failure to a layer and a mode — with a replay that proves the fix.

DeterministicModel-agnosticFailure attributionReproducible
No LLM in the scoreSandboxed executionAttributed tool failures carry proof

latest attributed run

Organizer-run

itsdangerous-future-signature

python (chaos: failing_tools)

anthropic/claude-sonnet-4.5

0.0

Attributed

harness × recovery

apply_patch failed (EIO: intercepted tool failure (injected)) and was not recovered

proof: 0.0 → 100.0 after retry at step 2

21 results in storeOpen result →

\ THE PROBLEM \ \ 01 \

A trace tells you what happened. It does not tell you why.

Developers stare at a 47-step trace and guess which layer failed. The model? The harness? A tool? The specification itself? Every guess costs another full run.

the hand-offs where signal is lost

01SPECsignal lost
02MODELsignal lost
03HARNESSsignal lost
04TOOLsignal lost
05ENVsignal lost

The agent sees an error, retries the wrong thing, or silently drops the result. The artifact fails, but the cause is not in the diff.

01

Model blamed for harness bugs

A swallowed tool error looks like bad reasoning when the harness never retried.

02

Tool failures look like decisions

A transient EACCES and a wrong design choice produce the same failed artifact.

03

No way to test a fix

Re-running the whole agent to test one policy change is slow, noisy, and expensive.

\ THE SOLUTION \ \ 02 \

A deterministic benchmark with an attribution layer.

The score is the substrate. The attribution is the product. Every failure gets a cell in a 5×4 taxonomy and a counterfactual replay that crosses the pass threshold.

01

Deterministic scoring

Declared checks and static analysis. No LLM in the loop; every point traces to a rule.

SCORE

02

Counterfactual attribution

Layer × mode with evidence, then a replay that changes one thing and proves the delta.

ATTRIBUTE

03

Reproducibility

Versioned tasks, rulesets, adapters, and a repository fingerprint on every row.

REPLAY

attribution card

FAILharness × recovery

evidence step 2, write_file → EACCES, no retry

proof replay with retry policy → PASS

delta 0.0 → 100.0

scope intervention at 2, replay 3–7, static scoring

5×4taxonomy cells
0model calls in replay
View the real proof →

\ HOW IT WORKS \ \ 03 \

Five steps. One reproducible record.

01

Spec

A versioned task pins the spec, ruleset, scoring config, and starting files.

task 1.0.0 · ruleset 1.0.0

02

Run

The agent executes in Docker; model calls pass through one allowlisted proxy.

trace 54 events · 7 tools

03

Score

Static checks and structural analysis produce the artifact score.

score 0.0 · test failed

04

Attribute

Rules map the failure to layer × mode with step-level evidence.

harness × recovery

05

Replay

A recorded intervention re-runs the tail and proves pass/fail.

0.0 → 100.0 · PASS

\ MEASUREMENT \ \ 04 \

Six metrics. Four chaos modes. One reliability profile.

The track asks for six metrics and resilience testing. Those are the substrate. Attribution is the layer above them: it names where the loss entered and proves a fix without guessing.

Artifact score

0–100, severity weighted

Steps

model + tool steps

Tool calls

count + error rate

Tokens

prompt + completion

Cost

provider usage

Duration

wall clock

tool_error

One write fails with EACCES; the harness either retries or loses the file.

failing_tools

The real-task apply_patch is intercepted; the replay retries it.

latency

Every model call gets a fixed delay; the stopping policy is tested.

context_pressure

Context is truncated and the step budget shrinks; completeness is tested.

\ LEADERBOARD \ \ 05 \

Reliability, not a single lucky run.

View full leaderboard →
Top benchmark results with reliability
#AdapterTaskModelScorePass@kReliability
1OpenAI Codex CLI (openai/gpt-5.1-codex)is-hexadecimal-single-charopenai/gpt-5.1-codex100.0100%
1 runs
2Aider 0.86.2 (anthropic/claude-sonnet-4.5)is-word-character-single-charanthropic/claude-sonnet-4.5100.0100%
1 runs
3OpenAI Codex CLI (openai/gpt-5.1-codex)is-number-reject-hexopenai/gpt-5.1-codex100.0100%
1 runs
4OpenAI Codex CLI (openai/gpt-5.1-codex)is-whitespace-single-charopenai/gpt-5.1-codex100.0100%
1 runs

\ ATTRIBUTION \ \ 06 \

Every failure named. Every fix proven.

An attribution without a counterfactual is an opinion. Orivoxx stores the baseline trace, the intervention, and the replay score. If no intervention crosses the threshold, the result says so.

Open a replay proof →

proof

Sufficient

0.0 → 100.0

1baseline trace
2intervention at step 2
3replay remaining steps
4deterministic score
PASS · harness × recovery · no model call in replay

\ FOR DEVELOPERS \ \ 07 \

Run it, attribute it, replay it.

The organizer CLI uses the same deterministic engine as the web app. Real adapters run in Docker; the public app reads the immutable result.

Adapter guide →

$ orivoxx bench --task taskboard --chaos tool_error

run 0.0 · harness × recovery · proof 100.0

$ orivoxx replay --result res_64068a

intervention retry_policy@2 · replayed 6 · PASS

$ orivoxx diff --a res_64068a --b res_df4fc5

score +100.0 · real test passes · no model in score

\ FAQ \ \ 09 \

Is this another trace viewer?+

No. A trace viewer shows steps. Orivoxx names the layer and mode, then proves the attribution with a replay.

How is attribution verified?+

The proof is a counterfactual replay that crosses the pass threshold. If no intervention works, the result says unattributed.

Does it work with any framework?+

The adapter contract is framework-agnostic: one argv command inside the sandbox. Python and Aider ship today.

What is deterministic?+

The score, the rules, the replay, and the attribution. No model output enters the score path.

What if the model provider is down?+

Stored results stay available. The score path makes no model call; a missing trace is reported as unavailable.

What can't it attribute?+

Failures outside the trace, unseen conditions, and cells the ruleset does not cover. Recall is published, not assumed.

Benchmark the agent. Prove the failure.

is: add an email predicate and EmailString type is waiting. Run the baseline, inject a fault, and open the replay.

Open source · MIT · Benchmark the agent, not just the model