Build-Bench

Build-Bench Challenge

Repair real cross-architecture package build failures with autonomous LLM Agents.

Cloud, edge, and emerging platforms increasingly span heterogeneous instruction set architectures, making software portability a growing engineering challenge. Architecture-specific code, dependencies, compilers, build options, and packaging logic can cause a package that builds on one architecture to fail on another. Build-Bench Challenge turns these failures into an executable, benchmark-driven competition for LLM-based repair Agents that generalize across packages, failure types, architectures, and migration directions.

Teams submit a runnable repair Agent rather than Case-specific patches. Organizers execute each qualified Agent on controlled source-to-target migration Cases, derive a canonical patch from its modifications, and reapply the patch to a clean copy of the package. A Case is counted as successfully repaired only when the patch complies with competition policy and the official target-architecture build completes with the expected package artifacts.

Evaluation scope: x86_64 ↔ aarch64 · x86_64 ↔ riscv64 · aarch64 ↔ riscv64

A repair Agent transforms a broken software package into a validated build artifact
Get the Starter Kit Explore the Challenge
Official competition
Accepted to the ICSE 2027 Competition Track View official track page
Organized by
Nankai University
Microsoft

Supporters

  • MeituanModel API support
  • Computer Network Information Center, Chinese Academy of SciencesCompute infrastructure support

What is the challenge?

Modern software increasingly needs to run across heterogeneous instruction set architectures (ISAs). A package that builds successfully on one architecture may fail after migration because of architecture-specific dependencies, compiler behavior, build configuration, or packaging logic.

Build-Bench Challenge asks teams to build and submit a runnable LLM-based repair Agent, rather than Case-specific repair patches. The competition covers bidirectional migrations among x86_64, aarch64, and riscv64 and evaluates whether Agents can generalize across packages, failure types, architectures, and migration directions.

For each Case, the Agent enters a prepared package workspace with source and target architecture metadata and initial target-build failure evidence. During the run, it may inspect the available package and build context, use tools permitted by the competition runtime, and iteratively modify only permitted package files.

After the Agent finishes, organizers derive a canonical patch from the Agent's final worktree, apply it to a fresh copy of the Case, and rebuild the package in the official target-architecture environment. A Case is successfully repaired only when the canonical patch applies cleanly, the official build succeeds, and the expected package artifacts are produced and verified. Repairs are judged by verified executable outcomes, not by similarity to a reference patch. The overall workflow is illustrated below.

Build-Bench Challenge evaluation workflow, from Case input and Agent execution to patch canonicalization, clean rebuilding, and scoring
The workflow summarizes how each submitted Agent is executed, converted into a canonical repair, and independently verified through a clean target-architecture build.

Build-Bench Challenge builds on our prior study accepted for publication in ACM Transactions on Software Engineering and Methodology (TOSEM) [1], which established the original cross-architecture package repair task and executable evaluation workflow. The competition extends this foundation to bidirectional migrations among x86_64, aarch64, and riscv64, with 200 public Development Cases and 1,000+ hidden evaluation Cases. It also introduces a standardized Agent interface and organizer-run evaluation for competition-scale assessment of Agent generalization.

Build your Agent

How the competition works

  1. Develop — Build and test your Agent with the Starter Kit and public development Cases.
  2. Qualify — Submit an Agent version and pass the Hosted Smoke Test.
  3. Compete — Freeze a qualified version for organizer-run evaluation on hidden Cases and leaderboard ranking.

How is performance scored?

Qualified Agents are evaluated on the official hidden Case set under the same runtime and resource constraints.

The primary ranking metric is Verified Build Success Rate — the percentage of evaluated Cases for which the canonical repair applies cleanly, the official target-architecture build succeeds, and the expected package artifacts are produced and verified.

Execution Time and officially recorded Token Usage are reported separately as secondary efficiency metrics.

Read the Full Competition Rules

Awards & Recognition

Winning teams will receive competition certificates and additional recognition associated with the ICSE 2027 Competition Track.

Timeline

  1. Release

    Website beta and initial documentation

  2. Pilot

    Invited pilot

  3. Policy freeze

    Rules and resource limits frozen

  4. Opening

    Public development and validation open

  5. Deadline

    Team registration closes

  6. Deadline

    Final Agent version freeze

  7. Results

    Final results published

References

  1. Chenyu Zhao, Shenglin Zhang*, Zeshun Huang, Weilin Jin, Yongqian Sun, Dan Pei, Chaoyun Zhang, Qingwei Lin, Chetan Bansal, Saravan Rajmohan, Minghua Ma. Can Language Models Go Beyond Coding? Assessing the Capability of Language Models to Build Real-World Systems. ACM Transactions on Software Engineering and Methodology (TOSEM), 2026. (CCF A) [paper]
  2. Chenyu Zhao, Minghua Ma*, Shenglin Zhang, Zeshun Huang, Yongqian Sun, Chetan Bansal, Saravan Rajmohan, Dan Pei. EvidenT: An Evidence-Preserving Framework for Iterative System-Level Package Repair. ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2026. (CCF A) [paper]