Skip to content

counterfactual proof · python (chaos: failing_tools)

harness × recovery

retry the intercepted apply_patch once at step 2

Pass

baseline

0.0

failed

counterfactual

100.0

3 replayed steps

intervention

retry_policy

at step 2 · threshold 70

Baseline trace

Steps before 2 are byte-identical in the replay. The highlighted step is where the intervention applies.

  1. —Run startedpython-openrouter 1.0.0 · run 0
  2. —Chaos injectedfailing_tools · the next apply_patch call is intercepted with EIO
  3. 1Model call startedanthropic/claude-sonnet-4.5 · messages 34de7fc9…
  4. 1Model call finished16246 in / 4276 out · $0.11288 · stop
  5. 2Tool call startedapply_patch · files=src/itsdangerous/exc.py,src/itsdangerous/__init__.py,src/itsdangerous/timed.pyredacted
  6. 2Tool call finishedapply_patch · error · EIO: intercepted tool failure (injected)redacted
  7. 2Step completed1 ms
  8. —Run finishedfailed · 2 steps · 1 tools · 20522 tokens · $0.1129

Deterministic scoring note

Replay reconstructs the repository from the recorded starting files, applies the intercepted action at step 2, reinstalls the task's declared dependencies, and re-runs its real test command inside the same sandbox. It makes no model call. The scoring engine never executes produced code.