Participate | Build-Bench Challenge

01 / Start

Quick start

Choose the coding-agent path for tools such as Codex, Claude Code, Cursor, or Copilot, or follow the same workflow manually. Both paths produce the same submission ZIP and are evaluated identically.

Recommended for coding assistants

Coding Agent Quick Start

Open an empty development directory in your coding assistant and give it the instruction below. This path clones the official repository directly and does not require downloading the Starter Kit ZIP.

View the Agent SDK and baseline repository

Clone https://github.com/AIOps-Lab-NKU/BuildBench-Agent-Baseline.git into BuildBench-Agent-Baseline. Enter BuildBench-Agent-Baseline/starter-kit. Read AGENTS.md before changing files. Bootstrap my-agent, implement the Repair Agent only in the generated Agent directory, then run the readiness workflow. Explain any failed check before changing configuration.

The coding assistant should follow the current contract in AGENTS.md and run:

git clone --depth 1 https://github.com/AIOps-Lab-NKU/BuildBench-Agent-Baseline.git
cd BuildBench-Agent-Baseline/starter-kit
./bb bootstrap my-agent --json
# After implementing agents/my-agent/src/
./bb ready --agent ./agents/my-agent --json

bootstrap checks the local environment and creates agents/my-agent/ without silently overwriting existing work.
ready snapshots that Agent, runs the released Example Cases and submission checks, and packages only a passing version.
Completion requires a JSON result with "status": "succeeded", an existing ZIP, and a matching SHA-256.

Expected: dist/agent-submission.zip is ready to upload. The coding assistant should report changed files, checks run, test results, ZIP path, and SHA-256.

bootstrap and ready provide the high-level coding-assistant workflow; the equivalent granular commands are documented in Manual Quick Start below.

Manual Quick Start

You need a Linux or WSL2 shell, Git, and Docker Engine 24+ or Docker Desktop with Linux containers. The following granular commands expose every preparation step for manual use and debugging.

01
Get the Starter Kit
Download the current ZIP from Data & Downloads, extract it in a Linux or WSL2 shell, and enter the versioned directory.
```
KIT_VERSION=0.1.0-rc.3
unzip "buildbench-starter-kit-${KIT_VERSION}.zip"
cd "buildbench-starter-kit-${KIT_VERSION}"
```
Configure
No project configuration is required at this step. Keep the entire extracted directory together.

Verify
Run pwd and ls; all later ./bb commands must be executed from this directory.

Expected: the current directory contains bb, agents/, and example-cases/.
02
Run the official demo
Check Git, Docker, and the versioned images, then run the complete failure-to-success example.
```
./bb doctor
./bb demo
```
Configure
Use the image references pinned by the release. If the organizer provides a registry mirror, set BB_AGENT_IMAGE, BB_VALIDATOR_IMAGE, and BB_EXAMPLE_ASSETS_IMAGE before running doctor.

What it runs
doctor performs environment checks only. demo uses the official Example Agent and the self-contained hello Case; it does not use your Agent code.

Expected: Final validation succeeded; evidence is saved under runs/demo/.
03
Create your Agent
Copy the managed Python template into a new editable Agent directory.
```
./bb init my-agent
```
Argument
my-agent becomes both the directory name and default agent.name. Use 2–64 lowercase letters, digits, or hyphens.

Configure
Edit repair logic under src/, pin Python dependencies with == in requirements.lock, and change name, version, or entrypoint in agent.yaml.

Expected: a new Agent is created at agents/my-agent/; existing directories are never overwritten.
04
Develop and test your Agent
Implement the repair logic in agents/my-agent/src/, then run every bundled Example Case.
```
# Edit agents/my-agent/src/main.py
./bb test --agent ./agents/my-agent
```
Input
--agent must point to the Agent directory that contains agent.yaml. The command runs each ID listed in example-cases/cases.txt; release v0.1.0-rc.3 contains one local Example Case, hello.

Agent behavior
The entrypoint declared in agent.yaml runs once per Case. It reads /workspace/input, edits /workspace/work/repo, and writes /workspace/output/agent-result.json.

Expected: per-Case evaluation artifacts, logs, and validation results are written under runs/tests/my-agent/.
05
Check and package
Validate the manifest, entrypoint, dependencies, and files before creating a deterministic upload bundle.
```
./bb check --agent ./agents/my-agent
./bb package --agent ./agents/my-agent
```
Check
Rejects missing required files, unsupported manifest fields, invalid entrypoints, unpinned dependencies, generated outputs, caches, symbolic links, and likely credentials.

Package
Writes dist/agent-submission.zip by default. Use --output path/name.zip only when you need a different local filename.

Expected: both checks pass and the command prints the ZIP path, size, and SHA-256.
06
Upload your Agent
Open My Submissions, choose Make new submission, and upload the generated ZIP.
```
ls -lh dist/agent-submission.zip
```
Upload
Select the ZIP produced by ./bb package, not the whole Starter Kit directory and not a ZIP you assembled manually.

After upload
The platform stores an immutable Agent version, performs static checks, and then runs the Hosted Smoke Test. Full Evaluation starts only after you explicitly select a qualified version.

Expected: a new immutable Agent version appears in the submission table and enters the Checking state.

Different versioned Case sets are used for local examples, development, the Hosted Smoke Test, and Full Evaluation. See Data & Downloads for their scope and release status.

02 / Package

Agent package

Upload one ZIP archive with a shallow, inspectable root. The platform reads the manifest before building the Agent runtime.

agent-submission.zip
├── agent.yaml
├── src/
├── requirements.lock
└── README.md

agent.yaml: Required. Declares Agent identity, managed runtime, entrypoint, and protocol version.
src/: Required. Contains the Agent implementation and the declared Python module or script.
requirements.lock: Required. Declare every third-party Python dependency with an exact == version.
README.md: Required in v0.1. Describes how the Agent works and any participant-facing notes.

Minimum `agent.yaml`

The manifest declares how the platform builds and starts the Agent. Version 0.1 supports the managed Python 3.11 profile; the Starter Kit validates this exact contract before packaging.

schema_version: "0.1"

agent:
  name: "example-agent"
  version: "1.0.0"

runtime:
  type: "managed"
  profile: "python-3.11"

entrypoint:
  - "python"
  - "-m"
  - "src.main"

protocol:
  version: "0.1"

03 / Runtime

Runtime interface

Each Case starts in a fresh workspace. The original input remains unchanged while the Agent works on a writable copy.

Execution contract

Invocation: The platform starts the entrypoint list from agent.yaml once for each Case, with the Agent bundle as the process working directory.
Workspace variable: BB_WORKSPACE=/workspace. Resolve all Case input, worktree, and structured output paths from this root.
Completion: For the current v0.1 protocol, exit with code 0 and write /workspace/output/agent-result.json with status: "completed". A non-zero exit is an Agent error.
Diagnostics: Write human-readable progress to stdout or stderr. Do not place secrets in logs; both streams are retained by the platform.

Workspace

/workspace/
├── input/            # read-only
├── work/
│   └── repo/         # writable
└── output/           # writable

Read input/.It contains the task metadata, initial failure evidence, and original package tree.
Modify work/repo/.Its final file state when the Agent exits is the official repair.
Use stdout and stderr for logs.The platform captures both streams for diagnostics.
Write agent-result.json.Place machine-readable completion status and diagnostics in output/; this file does not define the repair.

Repair output contract

The official repair is the final file state of /workspace/work/repo when the Agent exits. Within permitted paths, the Agent may edit or rewrite existing files, create new files, delete files, or generate and apply patches internally. Any internally generated patch must be applied to the worktree before exit. A patch written only to /workspace/output is not treated as the official repair.

{
  "schema_version": "0.1",
  "status": "completed",
  "message": "Repair attempt completed."
}

agent-result.json reports completion status and diagnostics, but it does not define the repair itself. modified_paths, if included, is advisory only. The evaluator independently determines the actual modifications by comparing the original and final worktrees.

Build feedback

Planned feature. Bounded hosted build feedback will be introduced in a later protocol release. Its CLI, limits, and response schema are not part of Starter Kit v0.1.0-rc.3.

Canonical patch

After the Agent exits, the evaluator compares the original and final worktrees and automatically derives a canonical repair.diff from all actual worktree changes. The canonical repair.diff uses Git extended unified-diff format, captures supported text changes, file creation and deletion, and file-mode changes, and disables rename detection for deterministic canonicalization. Participants do not need to generate or submit repair.diff. The evaluator replays this diff on a fresh Case and performs the official target-architecture build verification. See Evaluation and scoring for the complete verification rules.

04 / Qualify

Test and qualify

Test the exact Agent version you intend to evaluate. Local checks catch packaging errors; the Hosted Smoke Test checks that the same bundle can run under the competition protocol.

Before Full Evaluation

Before spending a Full Evaluation attempt, run ./bb check, test the Agent on all released Example Cases, package that exact version, upload it, and pass the Hosted Smoke Test. Full Evaluation begins only after you explicitly select the qualified immutable version.

Testing locally

Local testing should confirm that the submission contract works before organizer resources are used. Check all of the following:

The ZIP opens with agent.yaml and src/ at its root.
The manifest passes ./bb check and declares one valid managed-Python entrypoint.
The entrypoint starts without interactive input, writes the required structured result, and exits with code 0 when it completes normally.
/workspace/input remains unchanged; repository edits are confined to /workspace/work/repo.
/workspace/output/agent-result.json contains schema_version: "0.1" and a supported status.
The Agent completes at least one released local Example Case.

Release candidateStarter Kit v0.1.0-rc.3 provides the local Runner, managed-Python template, bundled Example Cases, conformance checks, and deterministic packaging. Public Development Cases support broader Agent development and local testing. See Data & Downloads for current downloads, versions, and release status.

Hosted Smoke Test

The Hosted Smoke Test uses the same Agent Runner, workspace layout, and status schema as Full Evaluation, but runs only a small set of lightweight qualification Cases. It is intended to expose missing dependencies, invalid entrypoints, permission errors, and malformed output before Full Evaluation.

Hosted Smoke Test results include more detailed logs and diagnostics than the leaderboard. They do not contribute to the official score, and passing the Hosted Smoke Test does not guarantee success on the full Case set.

05 / Rules

Rules & Policies

Before uploading an Agent, make sure your Team understands these essential participation rules.

Participating Agents must use an LLM as the foundation model.
Case-specific answers or precomputed Case-specific repair patches are prohibited.
All intended repair changes must be applied to /workspace/work/repo.
External models, APIs, tools, and network access must comply with the competition policy.
Submissions must not contain secrets, hidden Case information, or evaluation answers.
Only qualified immutable Agent versions may be selected for Full Evaluation.
Official results are determined by organizer-run evaluation under the published runtime and resource constraints.

Read the Full Competition Rules

06 / FAQ

Common Questions

Short answers to the questions participants most often ask while preparing an Agent.

Does my Agent need to use an LLM?: Yes. Participating Agents must use an LLM as their foundation model and may combine it with retrieval, static analysis, log processing, search, or other permitted tools.
What do I need to run the Starter Kit?: A Linux or WSL2 shell, Git, and Docker Engine 24+ or Docker Desktop with Linux containers are required.
Do I need sudo privileges?: No. The Starter Kit does not require sudo itself, but your user account must have permission to run Docker containers.
Can I use external models or APIs?: Only when their use complies with the published runtime, network, disclosure, and competition policies. Never include API keys or other secrets in the submission bundle.
What is the Hosted Smoke Test?: It verifies that the uploaded Agent can run under the competition protocol before Full Evaluation. It does not contribute to the official score.
How is my Agent evaluated?: Repairs are verified through a clean target-architecture build. Verified Build Success Rate is the primary ranking metric.

View all FAQs

07 / Final

Final checklist

Use this checklist on the exact immutable Agent version that you intend to upload and evaluate.

Final submission checklist

Required files exist at the ZIP root.
./bb check passes.
The entrypoint starts non-interactively and follows the declared runtime contract.
All dependencies are exactly pinned.
No secrets, caches, precomputed Case-specific repair patches, or run artifacts are included.
All intended repair changes are applied to /workspace/work/repo.
No intended repair is left only as an unapplied patch in /workspace/output.
agent-result.json follows protocol v0.1.
The uploaded version passes the Hosted Smoke Test.
The intended immutable version is selected for Full Evaluation.

Runtime and policy

The Agent runs as a non-root user in an isolated runtime, /workspace/input is read-only, and the Docker Socket is not provided. Remaining network, resource, quota, and submission policies will be published before submissions open.