Can Language Models Go Beyond Coding? Assessing the Capability of Language Models to Build Real-World Systems
1Nankai University · 2Peking University · 3Tsinghua University · 4Microsoft
*Corresponding author
Build-bench evaluates whether language-model agents can diagnose, repair, and verify real-world software packages that fail during migration between x86_64 and aarch64. It combines reproducible cross-ISA failures, standardized MCP-style tool orchestration, iterative build feedback, and executable verification on Open Build Service.
Build success rates and efficiency under the Build-bench protocol. The table reports the main full-file repair setting and includes Qwen2.5-3B-Instruct as a lightweight reproducible SLM baseline.
| Rank | Model / Agent | x86_64 → aarch64 (%) | aarch64 → x86_64 (%) | Avg. Success (%) | Avg. Time (min) | Avg. Tokens (K) |
|---|---|---|---|---|---|---|
| 1 | GPT-5 | 63.19 | 29.52 | 46.36 | 24.87 | 1674.79 |
| 2 | GPT-5-mini | 28.83 | 26.67 | 27.75 | 14.09 | 1789.28 |
| 3 | GPT-4o | 13.50 | 12.38 | 12.94 | 5.88 | 577.89 |
| 4 | Qwen3-max | 17.18 | 5.71 | 11.45 | 31.58 | 432.28 |
| 5 | Claude Sonnet 4.5 | 9.82 | 5.71 | 7.77 | 5.40 | 330.88 |
| 6 | DeepSeek V3 | 7.98 | 3.81 | 5.90 | 15.32 | 340.28 |
| 7 | Qwen2.5-3B-Instruct | 4.91 | 1.90 | 3.41 | 25.99 | 482.86 |
Avg. Success is the unweighted mean of the two directional success rates. The main protocol uses Nmax = 3 repair iterations and Tmax = 20 tool calls per iteration.
Most software-engineering benchmarks evaluate source-level coding, issue resolution, or test-passing behavior in homogeneous environments. Build-bench targets a different capability: repairing software packages whose build breaks when migrating across instruction set architectures.
Each task starts from a package that builds successfully on a source architecture but fails on the target architecture. The agent must inspect the package, localize the build failure, modify code or configuration files, upload the repaired package to Open Build Service, and use the executable build result as feedback.
A repair is counted as successful only when the package rebuilds successfully on the target architecture.
The 268 failures are grouped into five expert-validated classes that cover the build process from environment setup to runtime validation.
Missing macros, incompatible toolchains, unresolved dependencies, invalid compiler flags, or duplicate arguments that prevent configuration.
Repair signal: dependency declarations, architecture conditions, package metadata, and compiler setup.
Build-system incompatibilities, linker-level failures, type mismatches, missing headers, language-standard conflicts, or warnings promoted to errors.
Repair signal: source files, build scripts, toolchain behavior, and compiler diagnostics.
Missing binaries, unreferenced documentation, failing RPM scripts, failed install targets, duplicate installs, or policy violations.
Repair signal: install paths, spec sections, generated artifacts, and packaging rules.
Functional assertion mismatches, restricted runtime environments, missing test dependencies, crashes, timeouts, or resource-related failures.
Repair signal: test logs, runtime behavior, dependency setup, and architecture-specific assumptions.
Host, VM, or virtualization interruptions that terminate the build outside ordinary source-level repair.
Repair signal: external build-service state and reproducibility checks.
Complete source package, source archive, specification and metadata files, build scripts, and architecture-specific failed build logs.
Structure extraction, file-content extraction, decompression, compression, content modification, OBS upload, and build-result checking through a unified MCP-style interface.
Rebuild the updated package on OBS and return the latest build result, build log, and previous repair content for the next iteration when needed.
Build success rate, average repair time, average token consumption, and tool invocation behavior under a fixed iteration and tool-call budget.
The model regenerates the complete faulty file while preserving structure, style, and necessary context.
The model emits line-level unified diffs that are automatically applied to the source tree.
GPT-5 reaches the best full-file success rate, repairing 63.19% of x86_64 → aarch64 failures and 29.52% of aarch64 → x86_64 failures. All models still leave substantial headroom.
Bare single-shot repair performs poorly: GPT-5 reaches only 6.13% in the forward direction without tool use and iterative feedback, compared with 63.19% in Build-bench.
GPT-5 improves from 36.81% after the first iteration to 63.19% after the third in x86_64 → aarch64 migration.
Patch Generation reduces GPT-5 forward repair time from 31.18 to 8.93 minutes and token usage from 1830.91K to 761.88K, while Full File Generation remains more robust for broader consistency.
Models can miss the causal error among noisy compiler output, repeated dependency traces, and downstream packaging messages.
Some repairs are syntactically plausible but incomplete, especially when coordinating several files or regenerating long configuration content.
Agents sometimes modify decompressed files but upload stale or unrecompressed packages, leading to invalid OBS submissions.
Some models repeatedly check build results without substantive changes, increasing runtime without improving success.
Architecture migration often requires synchronized changes across spec files, source code, compiler flags, and generated artifacts.
This breakdown shows that the corpus taxonomy is not only descriptive: repair performance varies by failure cause and by editing granularity. Patch Generation is often stronger for localized preparation or packaging fixes, while Full File Generation is more reliable for broader contextual regeneration in several forward-migration categories.
Tasks involve package-level failures caused by heterogeneous ISAs, toolchains, dependencies, build scripts, packaging rules, and long build logs.
A task succeeds only when the modified package is rebuilt successfully on the target architecture through Open Build Service.
Repair attempts are validated by executable rebuilds, not by textual similarity, static prediction, or single-file tests.
Results expose model gaps in log comprehension, procedural tool use, cross-file reasoning, and architecture-specific adaptation.
@misc{zhao2025languagemodelscodingassessing,
title={Can Language Models Go Beyond Coding? Assessing the Capability of Language Models to Build Real-World Systems},
author={Chenyu Zhao and Shenglin Zhang and Zeshun Huang and Weilin Jin and Yongqian Sun and Dan Pei and Chaoyun Zhang and Qingwei Lin and Chetan Bansal and Saravan Rajmohan and Minghua Ma},
year={2025},
eprint={2511.00780},
archivePrefix={arXiv},
primaryClass={cs.SE},
url={https://arxiv.org/abs/2511.00780}
}