Submit an Agent, not precomputed answers
A team supplies a reproducible Agent that can consume the prescribed case input and emit the prescribed repair output. The competition harness invokes that Agent on organizer-controlled validation and hidden test cases.
The Agent may internally use prompting, retrieval, static analysis, build-log parsing, tool calls, or iterative repair loops. These are implementation choices inside the same submission model, not separate submission modes.
The repair patch remains the common artifact that the evaluator applies and builds. It is an Agent output, not a second way to enter the competition.
From Agent artifact to leaderboard score
- 1Team submits Agent
The platform validates metadata and stores an immutable, versioned entry.
- 2Harness starts runtime
The organizer launches the Agent in an isolated environment with the announced limits.
- 3Harness provides a case
The Agent receives a case directory and metadata through the versioned interface.
- 4Agent emits a repair
The Agent writes a unified diff and structured completion status to the output location.
- 5Evaluator builds package
The repair is applied to a clean copy and verified with the official target build.
- 6Results are aggregated
Valid case outcomes contribute to the submission score and leaderboard statistics.
One ZIP contains the complete case input
The harness will provide one read-only case.zip per case. The final starter kit will define the exact tree and JSON Schema; the current working layout is:
case.zip
|-- manifest.json
|-- source/
|-- spec/
|-- patches/
|-- metadata/
\-- logs/
\-- target-build.logPackage specifications, source archives, existing patches, auxiliary build-service files, and the failed log.
Case ID, package name, source architecture, target architecture, validator backend, and checksums.
Fixed CLI command, local HTTP API, or equivalent harness adapter.
Whether an Agent can request build attempts during one case, and which logs are returned after each attempt.
One ZIP returns the repair and status
The Agent will return one result.zip. A unified diff remains the repair artifact because it is inspectable, auditable, and can be applied to a clean package copy.
result.zip
|-- result.json
\-- repair.diff
{
"case_id": "example-case",
"status": "repaired",
"patch_path": "repair.diff"
}The sample communicates the shape of the contract, not guaranteed names or values. The released JSON Schema and conformance tests will be authoritative.
Expected result classes
A candidate diff is available for policy checks and executable validation.
The Agent completed but intentionally emitted no patch for the case.
The Agent crashed, timed out, or returned malformed output; final scoring semantics are TBA.
Reproducible packaging is required; the vehicle is TBA
The Agent must be runnable without manual intervention and must expose one deterministic entrypoint. The team has not yet chosen whether the official artifact will be a container image, a source repository plus lockfile, or another packaged runtime.
| Decision | Current status |
|---|---|
| Submission artifact | Open Container image, repository bundle, or platform-managed package |
| Entrypoint | Open Fixed CLI or local API |
| Architecture of Agent runtime | Open Native, emulated, or platform selected |
| Dependency installation | Open Prebuilt only or bounded setup phase |
| Versioning | Intended Immutable artifact digest per leaderboard entry |
Policies that must be frozen before submissions open
Per-case wall time, maximum repair attempts, tool calls, and build requests.
CPU, memory, disk, accelerator availability, and concurrency.
Whether outbound access is disabled, allowlisted, or brokered only through organizer APIs.
Supported model providers, secret injection, accounting, and whether teams may bring their own keys.
Startup retries, transient API failures, malformed output, timeout, and evaluator fault handling.
Daily limits, artifact replacement, validation queue priority, and final-entry selection.
Organizer baseline plan: comparable baseline runs will use no more than two open-source model configurations and one closed model configuration, with versions and parameters frozen and reported. This does not restrict participant Agents unless the final Rules explicitly say so.
What participants will receive before launch
- Versioned Agent interface specification
- JSON Schema or equivalent validation contract
- Minimal conforming example Agent
- Local conformance and smoke-test command
- Development cases with expected evaluator outputs
- Published resource, model API, network, and failure policies