Task & Dataset | Build-Bench Challenge

Objective

Repair portability failures at system scale

Each task starts from a package that builds on a source architecture but fails when rebuilt on a target architecture. The Agent must inspect the supplied evidence and produce a focused repair.

GivenA reproducible failed package case

ProduceA valid repair artifact

PassThe official target build

The evaluator does not require a repair to match an organizer-authored reference patch. A repair succeeds only when it applies cleanly, respects the modification policy, and completes the configured build validator.

Case anatomy

What an Agent receives

Released cases will use a stable directory layout. The exact manifest schema and immutable paths will be versioned with the starter kit.

Package specificationsRPM .spec files or equivalent package metadata

Package artifactsSource archives, existing patches, service files, and auxiliary build-service files

Failure evidenceThe original failed build log from the target architecture

Architecture labelsSource architecture, target architecture, and validator backend metadata

Integrity metadataCase identifier, checksums, constraints, and paths that must remain immutable

Repair workflow

Inspect, diagnose, repair, verify

Build-Bench workflow from failed package context through agent repair to Open Build Service validation — The research workflow is being adapted into an organizer-run competition harness.

1
Inspect the package context
Read manifests, specifications, source files, build scripts, existing patches, and the failed log.
2
Diagnose the cross-architecture fault
Identify architecture assumptions, unavailable dependencies, compiler differences, packaging errors, or target-sensitive tests.
3
Generate a repair
Modify only permitted package files and emit the required repair artifact through the Agent interface.
4
Validate by building
The official evaluator applies the repair to a clean copy and rebuilds on the target architecture.

Research corpus

268 reproducible real-world failures

This preview follows the latest Build-Bench paper: 163 x86_64-to-aarch64 cases and 105 aarch64-to-x86_64 cases. The original 268 cases are being revalidated before the competition dataset is frozen.

163x86_64 → aarch64Paper-reported corpus

105aarch64 → x86_64Paper-reported corpus

Expansion status: 200 additional public package candidates have been discovered. They remain candidates, not accepted benchmark cases, until the original failure and repaired success are reproduced under a frozen environment.

Failure categories

Planned competition splits

Development, validation, and hidden test

Split	Visibility	Purpose
Development	Public	Local debugging, tutorials, and method development
Validation	Public inputs and scores	Leaderboard feedback during the public phase
Test	Held out	Organizer-run final ranking

Cases will be grouped by package so that near-duplicate packages do not appear across splits. Final counts and package assignments are not yet published.

Provenance and licensing

Release only what can be shared responsibly

The paper corpus originates from public openSUSE build scenarios. Dataset expansion is also investigating other distributions, package ecosystems, and ISA directions that can be reproduced in a containerized build environment.

Before release, organizers will audit license metadata, remove local credentials and personal paths, document redistribution constraints, and group semantic duplicates.

When redistribution is unclear, the starter kit may provide reconstruction scripts that retrieve artifacts from public package sources instead of distributing original archives.

Hugging Face dataset: the public repository link will be added after the first dataset version passes validation, licensing, and deduplication review.