1. Start and register
What is Build-Bench, and what does a Team submit?
Build-Bench evaluates runnable software repair Agents on real package build failures. A Team submits Agent source and a versioned runtime manifest; the organizers run that Agent independently on competition Cases. Teams do not submit pre-generated repair patches or a Docker image as their competition entry.
How does Team registration work?
One Team leader creates the account and enters the complete roster. A Team may contain up to five people including the leader. Every member email is required, and the same email cannot appear in another Team.
Where should a first-time participant begin?
Download the current Starter Kit, run its environment check and official demo, create an Agent from the template, and test that Agent locally before uploading it. The Submission Guide provides the copy-and-run commands.
Must an Agent use a large language model?
Yes. An LLM must serve as the Agent's foundation model. Teams may combine it with retrieval, static analysis, log processing, search, and other compliant tools, but the submission must remain a runnable Agent; Case-specific answer tables and precomputed Case-specific repair patches are prohibited.
2. Develop locally
What do I need to run the Starter Kit?
You need a Linux or WSL2 shell, Git, and Docker Engine 24 or later, or Docker Desktop using Linux containers. The Starter Kit does not require sudo privileges, but you must have permission to run Docker containers. Docker Desktop typically requires no additional sudo configuration.
Starter Kit v0.1.0-rc.3 includes the bb command, a managed-Python Agent template, an Example Agent, the hello Example Case, local checks, and deterministic packaging.
What can the Agent read, modify, and return?
The Agent reads task evidence from the read-only input directory, modifies only the writable package worktree, and may write machine-readable status to the output directory. It must follow the workspace paths and agent-result.json schema defined by protocol v0.1.
Can I submit a custom Docker runtime?
Not in Starter Kit v0.1.0-rc.3. The current submission contract supports the managed Python 3.11 profile. If additional runtime profiles are introduced, they will be published as a versioned protocol update rather than silently enabled.
Can the Agent request build feedback while it runs?
Not in the current Starter Kit release. Bounded hosted build feedback is a planned feature; its command, limits, and response schema will be introduced only in a later protocol release. Do not assume that bb-build is currently available.
Can an Agent call external model APIs or include API keys?
Do not include API keys, .env files, or other secrets in the uploaded bundle. The final network-access and organizer-managed credential policy will be published with the frozen rules and resource limits on August 31, 2026.
3. Upload and qualify a version
What file should I upload?
Upload the dist/agent-submission.zip archive produced by ./bb package. Do not upload the entire Starter Kit directory or assemble the competition archive manually. Every accepted upload is stored as an immutable Agent version with its own identifier and content digest.
What does the Hosted Smoke Test check?
It checks the uploaded bundle, entrypoint, dependencies, workspace permissions, and output contract on a small lightweight Case set. Passing it qualifies that immutable version for Full Evaluation, but produces no official score and does not guarantee success on the full Case set.
Can I upload a revised Agent or replace a running evaluation?
You may upload a revised bundle before the applicable deadline and within the published limits; it becomes a new immutable version. A later upload does not alter an active or completed evaluation. Full Evaluation begins only after the Team explicitly selects a qualified version.
4. Evaluation and scoring
How is a proposed repair judged?
The platform derives a canonical patch from the Agent's modified worktree, checks the permitted paths and repair policy, reapplies the patch to a clean Case, and invokes the official target-architecture Docker Validator. A repair succeeds because the genuine package build succeeds, not because it resembles a reference patch.
How is the competition score calculated?
Verified Build Success Rate is the primary metric: successful Cases divided by the official evaluation denominator. The exact denominator semantics, any tie-breaker, and remaining resource and rerun rules will be published in the frozen Rules on August 31, 2026.
How are failures, timeouts, and infrastructure errors handled?
Build failure, Agent error, timeout, and an invalid patch are unsuccessful Case outcomes. An organizer-controlled infrastructure error is handled separately: no partial score is published, and the affected work is reviewed or rerun under the competition procedure.
What is the difference between Smoke Test, Full Evaluation, and hidden evaluation?
Hosted Smoke Test is a small qualification run with detailed diagnostics and no score. Public-phase Full Evaluation measures a selected qualified version on the versioned validation set. Final hidden evaluation runs the Team's frozen final Agent on held-out organizer-controlled Cases and releases only the permitted aggregate results and diagnostics.
5. Data, dates, and results
Which Case sets are used, and how large is the benchmark?
Local examples, public development resources, Hosted Smoke Test, Full Evaluation, and final hidden evaluation use separate versioned Case sets. The published benchmark contains 268 x86_64 and aarch64 migration failures. The organizers aim to retain approximately 1,000 Cases for final hidden evaluation, subject to final integrity, licensing, deduplication, and reproducibility checks.
Where can I see status, results, and authoritative updates?
My Submissions shows uploaded Agent versions, qualification logs, Full Evaluation progress, and completed results. The Leaderboard shows published rankings. Versioned changes to dates, resources, rules, and evaluation policy appear on their corresponding competition pages; any official support channel will be identified on this website when available.
Still unsure? Use the linked detail page as the authoritative source. If an answer and a versioned rule or protocol differ, the latest published rule or protocol controls.