Skip to content

Benchmarks

Each task pins a specification, a ruleset, and a scoring configuration. The version triple is recorded on every result, so a score can always be reproduced against the exact rules it was measured with.

Task contract

  • PROJECT_SPEC.md is parsed deterministically; declared checks are the only source of verdicts.
  • Starting files are the only inputs. The produced repository is captured as bounded text and scored statically; produced code is never executed by the scoring engine.
  • Missing evidence is UNVERIFIED, earns zero, and is reported separately from FAIL.