Entering the competition
What exactly does a team submit?
One repair Agent that follows the versioned organizer interface. For each case, the working protocol supplies case.zip and expects result.zip, containing structured status and an optional repair.diff. Exact schemas and Agent packaging remain draft; see Agent Submission.
Are patch output and Agent container separate submission modes?
No. The competition now intends to use a single Agent submission model. A unified diff is the common repair output produced by the Agent and consumed by the evaluator. It is not a second competition entry path.
Must the Agent use an LLM?
The primary leaderboard is intended for LLM-based and agentic repair systems. Traditional non-LLM systems may be included as clearly labeled comparison baselines, subject to the final rulebook.
How large can a team be?
Team-size, affiliation, registration, and conflict-of-interest rules have not yet been announced. They will follow the ICSE Competition Track requirements and the released competition rulebook.
Runtime and API questions
Will the Agent be submitted as a Docker image?
Possibly, but that choice is not final. The organizing team still needs to choose the packaging format, entrypoint, runtime architecture, and dependency policy. The released starter kit will contain a minimal conforming Agent.
What input will the Agent receive?
The intended input includes a read-only case directory with package specifications, source archives, existing patches, auxiliary files, the original failed build log, architecture labels, checksums, and manifest metadata. Exact field names and paths are TBA.
What must the Agent return?
The planned common repair artifact is a unified diff plus a machine-readable completion status. The final schema, supported no-repair status, error fields, and output paths will be published as a versioned contract.
Can an Agent request more than one build attempt?
The research benchmark supports iterative repair, but the competition's maximum attempts, returned feedback, time budget, and tool-call limits are not final.
Can the Agent call external model APIs?
The model and network policy is still open. Organizers may disable general outbound access and provide allowlisted or brokered model APIs. Participants must not embed personal or production secrets in an Agent artifact.
Scoring and leaderboard questions
How is a repair judged?
The evaluator applies the Agent's repair to a clean case, checks modification policy, and runs the official target-architecture build validator. Textual similarity to an organizer patch is not the success criterion.
Are the current metrics final?
Build Success Rate is the working single primary ranking metric. Exact denominator semantics, failure handling, and any tie-breaker remain draft on the Evaluation page.
What happens if the Agent crashes or times out?
The final scoring semantics are TBA. The rulebook must distinguish Agent failure, malformed output, build failure, infrastructure failure, and cases canceled by the organizers.
Will model cost or token usage affect ranking?
This has not been decided. The organizer must determine whether Agent runtime, model cost, token use, build time, and repair iterations are ranking metrics, tie-breakers, or informational columns.
Why does the current leaderboard show model results?
Those rows reproduce research results from the Build-Bench paper so participants can understand the benchmark. They are not competition entries and will remain visibly separated from the future public competition ranking.
Cases, architectures, and OBS
How many Build-Bench cases are there?
The latest paper reports 268 reproducible cases: 163 for x86_64-to-aarch64 and 105 for aarch64-to-x86_64. Another 200 public package candidates are under validation and are not yet accepted benchmark cases. Final split counts will be frozen before launch.
What is OBS and why is it used?
Open Build Service is a platform for building packages across distributions and architectures. Build-Bench uses it as an executable validation backend: after a repair is applied, OBS rebuilds the package for the target architecture and reports whether the real build succeeds.
Can participants validate locally?
Yes. The working environment prioritizes a Docker-based local validator for reproducible development. OBS remains useful for research-corpus reconstruction and organizer-side evidence checks. The starter kit will document prerequisites and architecture-emulation limitations.
Will every package artifact be redistributed?
Only when licensing permits. For cases with unclear redistribution status, organizers may release scripts that reconstruct the case from public package sources.
Releases, deadlines, and updates
When will public validation open?
July 31, 2026 is an internal preparation checkpoint, not a participant deadline. The accepted proposal targets September 2026 for public validation, but the exact date will be aligned with the official ICSE 2027 schedule.
When is the final submission deadline?
The proposal targets a November 2026 freeze for hidden evaluation. The exact date, timezone, artifact selection procedure, and rerun window remain TBA.
Where will rule and evaluator changes be announced?
Changes will be versioned on the competition website and announced through the public support channel selected by the organizers. A frozen evaluator version will be announced before final submission.
Will teams be able to submit solution papers?
If the ICSE track invites solution papers, eligible teams are expected to need a valid final entry, rule compliance, system disclosure, and performance above an announced baseline. Track-chair policy and page limits will control the final process.