Evaluation | Build-Bench Challenge

Protocol status

Separate source material from final rules

Item	Basis	Status
Executable build as the success oracle	Build-Bench paper and accepted proposal	Selected
Build Success Rate as the single primary metric	Accepted proposal and organizer working decision	Selected
Failure categories and denominator semantics	Required for reproducible scoring	Team review
Tie-breaker and informational fields	Secondary to the primary metric	Open

Core principle

Executable outcomes are the oracle

A repair is successful only when it applies cleanly, respects competition policy, and the repaired package passes the official validator on the target architecture.

Primary ranking metric

SuccessRate=successfully built casesevaluated cases

Textual similarity to an organizer patch is not the criterion. A different repair may receive credit when it produces a compliant, successful build.

Evaluation harness

Seven observable stages

1
Validate entry
Verify the Agent artifact, configuration, version, and submission metadata.
Harness
2
Prepare clean case
Check case checksums and create an isolated working copy.
Harness
3
Invoke Agent
Provide standardized input and enforce runtime, API, network, and resource policy.
Agent
4
Collect output
Validate the completion status and repair artifact against the released contract.
Harness
5
Apply and audit repair
Apply the diff to a clean copy and reject forbidden path changes or build bypasses.
Evaluator
6
Run target build
Launch the official containerized validator on the target architecture under the published resource policy.
Validator
7
Record outcome
Write structured status, durations, patch statistics, and permitted logs.
Results

Reporting fields

One ranking score, with separate diagnostics

Role	Field	Working definition	Status
Primary ranking	Build Success Rate	Successfully built cases divided by valid evaluated cases	Selected
Diagnostic	Outcome categories	Successful build, build failure, invalid output, Agent failure, or organizer fault	Definition review
Diagnostic	Runtime and model use	Agent runtime, build time, token use, model/API cost, retries, and iterations	Informational
Diagnostic	Repair statistics	Patch size, modified files, and policy validation status	Informational
Secondary ordering	Tie-breaker	Applied only when primary scores are equal	To be announced

Diagnostic fields explain results but do not form a weighted multi-factor score. The final rulebook will state whether any one field is used only to break an exact tie.

Feedback policy

Public development without leaking the hidden test

Public validation

Actionable feedback

Aggregate leaderboard score
Per-case outcome on released validation inputs
Permitted build logs and evaluator status
Versioned evaluator and case manifests
Submission-frequency limit to reduce overfitting

Hidden final

Controlled disclosure

Organizer-run held-out cases
Frozen Agent artifact and evaluator version
Aggregate results after the deadline
Limited diagnostic categories
Audit and rerun of suspicious or affected entries

Open decisions

Items the organizing team must resolve

01

Denominator semantics Decide how Agent crashes, invalid output, organizer faults, and canceled cases affect evaluated-case counts.

02

Budget policy Freeze wall time, build attempts, iterations, tool calls, CPU, memory, storage, and concurrent cases.

03

Cost reporting Decide whether token use, model/API spend, Agent runtime, and build time are ranking metrics or informational columns.

04

Tie-break precision Define invalid-repair classes, patch-size calculation, file counting, and build-time aggregation.

05

Leaderboard cadence Set submission limits, queue rules, rerun policy, score visibility, and the final-entry selection deadline.

06

Reproducibility checks Define artifact retention, rerun tolerance, nondeterminism policy, and required disclosures.