Build autonomous LLM Agents that repair real package build failures across x86_64, aarch64, and riscv64.

Motivation and task

Cloud, edge, and emerging computing platforms increasingly rely on heterogeneous instruction set architectures. However, architecture-specific code, dependencies, compiler behavior, build options, and packaging logic can cause software packages to fail when migrated to a new architecture. Diagnosing these failures often requires expertise across source code, toolchains, packaging systems, and hardware architectures, making large-scale migration costly and time-consuming. Such platform migration needs are common across organizations, and with the growing adoption of cost-effective AI workloads, more teams are considering cross-architecture migration to optimize their infrastructure.

Build-Bench Challenge asks whether LLM-based repair Agents can automate this process reliably and generalize across packages, failure types, architectures, and migration directions. Each Case begins with a real software package that builds successfully on a source architecture but fails when rebuilt for a target architecture. The Agent must investigate the failure and modify the permitted package files so that the package can be rebuilt successfully for the target architecture.

Participants may design their Agent prompts, tools, and repair procedures freely. During each Case, the Agent must apply all intended changes directly to the permitted package worktree. It may edit individual files, rewrite complete files, create or delete files, or generate and apply patches internally.

After the Agent exits, the organizers compare the final worktree with the original Case and automatically derive a canonical diff that captures all worktree changes. This diff is then replayed on a clean copy of the Case for verification in the official target-architecture environment.

Goal

Repair a package that builds on the source ISA but fails on the target ISA.

Input

A prepared package workspace, source and target ISA metadata, failed-build evidence, and package build context.

Agent output

The final state of files within the permitted worktree, together with the required structured completion result.

Verification

A canonical patch replayed on a clean Case and evaluated by the official target-architecture build.

Evaluation scope

The challenge is designed to measure whether an Agent generalizes across software packages, failure types, instruction set architectures, and migration directions. It therefore evaluates all bidirectional migration pairs among the three supported ISAs:

x86_64 ↔ aarch64  ·  x86_64 ↔ riscv64  ·  aarch64 ↔ riscv64

Participants receive more than 200 public development packages for understanding the task and testing their systems. Final evaluation uses over 1,000 hidden packages drawn from broader software ecosystems and sources. Hidden evaluation is intended to discourage Case-specific rules and measure performance beyond packages seen during development.

Case inputs

At the beginning of each Case, the Agent receives the information needed to diagnose the initial target-architecture failure. Depending on the package, the workspace may contain source code, packaging specifications, metadata, build scripts, existing package-side patches, and other build-related files.

The standard Case input includes:

• a prepared package workspace;
• source and target architecture metadata;
• the initial failed target-build log; and
• the packaging and build context included with that Case.

The exact directory layout and runtime interface are defined in the versioned Submission Guide.

Agent behavior and constraints

The Agent may inspect the available package and build context, use relevant tools, analyze the failure, and modify files within the permitted package paths. It must complete the task autonomously during organizer-run evaluation.

All participating systems are evaluated under the same versioned execution contract. The Submission Guide defines the Agent package, workspace, entrypoint, permitted paths, and required outputs.

Repair output contract

The Agent must apply every intended repair directly to /workspace/work/repo. Within the permitted paths, it may edit existing files, create new files, or delete files. An Agent that generates a patch internally must apply that patch to the worktree before it exits. A candidate patch written only to the output directory is not treated as the official repair.

After completing its repair attempt, the Agent must write /workspace/output/agent-result.json. The result reports completion status and diagnostics, but it does not define the repair itself. Any declared modified_paths field is advisory; the evaluator independently determines the actual changes from the worktree.

Canonical patch

After the Agent exits, the evaluator compares the original and final worktrees and derives repair.diff in Git extended unified-diff format. This format records text changes, new and deleted files, and supported file-mode changes using paths relative to the package root. Rename detection is disabled during canonicalization so that renames are represented deterministically as a deletion and an addition.

diff --git a/package.spec b/package.spec
--- a/package.spec
+++ b/package.spec
@@ -42,1 +42,1 @@
-BuildRequires: old-dependency
+BuildRequires: portable-dependency

The public development environment is intended to match the official interface, but only results produced by the organizer-run evaluator count toward the leaderboard.

Evaluation and scoring

During official evaluation, organizers execute each qualified Agent on hidden Cases under fixed runtime and resource constraints. After each run, the evaluator captures the Agent's final changes and derives a canonical patch. The patch is then applied to a fresh copy of the Case.

The patched package is rebuilt in the official target-architecture environment. A Case is counted as successfully repaired only when all of the following conditions hold:

1. the canonical patch is generated successfully and applies cleanly to the fresh Case;
2. the official target-architecture build completes successfully; and
3. the expected package artifacts are produced and verified.

Repairs are judged by these verified executable outcomes, not by similarity to a reference patch. If the evaluator cannot derive or cleanly apply the canonical patch, the Case is recorded as invalid_patch and does not count as a successful repair.

Getting started

Use the Starter Kit and public development Cases to understand the workspace, test your Agent locally, and prepare a qualified submission. Before submitting, review the runtime interface, challenge rules, and evaluation protocol.

• Get the Starter Kit →
• Read the Submission Guide →
• Read Evaluation and Scoring →