Skip to content

Compare

Two results on one task. The same model slug means only the harness varies.

taskboardtask 1.0.0 · ruleset 1.0.0 · scoring 1.0.09 results on this task

A attribution

harness × recovery

proof 49.7 → 98.3

B attribution

none

no proof

Run metric deltas, B minus A
MetricABB − A
Steps70-7
Tool calls60-6
Tool error rate0.1670-0.167
Total tokens10,4870-10,487
Cost USD0.0040-0.004
Duration ms28,4880-28,488
Different modelThe harness and the model both differ, so the score gap is not attributable to one.

A

Organizer-run

Python agent loop (deepseek/deepseek-chat-v3-0324)

deepseek/deepseek-chat-v3-0324

49.7

Spec
33.3
Security
66.7
Maintainability
87.1
5 pass4 fail0 unverifiedcapped
Open result

B

Organizer-run

Aider 0.86.2 (openai/gpt-4o-mini)

openai/gpt-4o-mini

97.3

Spec
100.0
Security
100.0
Maintainability
82.0
9 pass0 fail0 unverified
Open result
Bench deltas, B minus A
BenchABB − A
Overall49.797.3+47.6
Spec33.3100.0+66.7
Security66.7100.0+33.3
Maintainability87.182.0-5.1
Requirement outcomes for both results
RequirementBenchAB
REQ-AUTH-001Project access must be owner-scopedspecFAILPASS
REQ-DEP-001No shell execution dependencysecurityPASSPASS
REQ-ERR-001Project API returns structured error responsesspecFAILPASS
REQ-PATH-001Projects API route existsspecFAILPASS
REQ-SEC-002No environment file in the repositorysecurityPASSPASS
REQ-SEC-003No dynamic code evaluation in the project routesecurityFAILPASS
REQ-TASKS-001Task list API route existsspecPASSPASS
REQ-TASKS-002Task list reads through the repository layerspecPASSPASS
REQ-UI-001Task list component existsspecPASSPASS