Benchmarks
Each task pins a specification, a ruleset, and a scoring configuration. The version triple is recorded on every result, so a score can always be reproduced against the exact rules it was measured with.
REAL REPO · is-email-predicate · v1.0.0
is: add an email predicate and EmailString type
The guard must validate a practical email shape while the public TypeScript surface gains a branded EmailString type.
- Difficulty
- Intermediate
- Real test
- node · MIT
- Versions
- r1.0.0 · c1.0.0
REAL REPO · itsdangerous-future-signature · v1.0.0
itsdangerous: add SignatureNotYetValid
A future-dated timestamp is currently reported as expired; it needs its own exception class, package export, and signer branch.
- Difficulty
- Advanced
- Real test
- python · BSD-3-Clause
- Versions
- r1.0.0 · c1.0.0
REAL REPO · slugify-max-length · v1.0.0
slugify: add a maxLength option
A new option must cap the final slug at a boundary, with the runtime and the published type declaration kept in sync.
- Difficulty
- Intermediate
- Real test
- node · MIT
- Versions
- r1.0.0 · c1.0.0
taskboard · v1.0.0
TaskBoard: owner-scoped project API
Build the project-access API for a small task board. Enforce ownership, return structured errors, read through the repository layer, and avoid shell-execution or dynamic-evaluation patterns.
- Difficulty
- Starter
- Requirements
- 9 · 6s / 3sec
- Versions
- r1.0.0 · c1.0.0
linkstash · v1.0.0
Linkstash: authenticated bookmark API
Build a bookmark API for a small authenticated product. Resolve the user, validate input with Zod, return structured errors, and avoid dynamic evaluation and shell-execution dependencies.
- Difficulty
- Intermediate
- Requirements
- 12 · 9s / 3sec
- Versions
- r1.0.0 · c1.0.0
Task contract
- PROJECT_SPEC.md is parsed deterministically; declared checks are the only source of verdicts.
- Starting files are the only inputs. The produced repository is captured as bounded text and scored statically; produced code is never executed by the scoring engine.
- Missing evidence is UNVERIFIED, earns zero, and is reported separately from FAIL.