Overview
What Orivoxx measures and why the score is deterministic.
Orivoxx benchmarks AI software-engineering agents: the model, the harness, the tools, the orchestration, and the stopping conditions as one configuration. A versioned task supplies a PROJECT_SPEC.md. The harness runs the agent in an isolated Docker sandbox, captures the produced repository, and the deterministic engine scores it on spec conformance, security patterns, and structural maintainability.
Every point traces to a named rule and evidence. The model is never in the score path. Optional explanations are a separate, explicit action and cannot change a number.
What is measured
- Spec conformance (60%): declared deterministic checks from the task ruleset.
- Security patterns (25%): absence of shell-execution dependencies, environment files, dynamic evaluation, and other declared rules.
- Structural maintainability (15%): import cycles, fan-out, orphan ratio, and module count from static analysis.
What is not measured
- Model quality in the abstract. Two harnesses on one model can score differently.
- Runtime performance or test execution. The scoring engine never runs produced code.
- Subjective style. There is no LLM judge and no preference model.