The accepted proposal specifies milestone months rather than final deadlines. Dates below should be used for planning, not as binding submission cutoffs.
Planned competition sequence
- In progress
Complete the first data, validator, and protocol pass
Finish the first audit of the original cases, package the first validated new cases, run the Docker validator proof of concept, and publish working input/output and evaluation drafts.
- Original 268-case audit status
- First validated expansion cases
- case.zip and result.zip draft
- Docker validator end-to-end record
- Website and Dataset Card draft
- Planned
Development set and baselines released
Release public development cases, conformance examples, evaluator documentation, and reproducible baseline Agents.
- Planned
Public validation leaderboard opens
Accept versioned Agent entries, run public validation cases, and return bounded diagnostic feedback.
- Planned
Final submission freeze
Freeze each team's selected Agent artifact, the interface version, evaluator, case manifests, and resource policy for hidden evaluation.
- Planned
Final results and participant reports
Complete hidden evaluation, audit finalist repairs, publish aggregate results, and collect eligible participant reports.
- Conference
Competition session and results presentation
Present the benchmark, leading systems, common failure modes, and final competition outcomes to the ICSE community.
What must be ready before public validation
Entrypoint, input/output schemas, packaging, conformance tests, and failure semantics are versioned.
Metrics, budgets, allowed APIs, network policy, tie-breakers, and submission limits are published.
Splits, licensing, checksums, immutable paths, and near-duplicate grouping are verified.
Example Agent submission, case execution, containerized build validation, structured result, and leaderboard update all succeed.
Freeze first, then run hidden cases
The final hidden evaluation is expected to finish within one day under the initial proposal target. The exact queue, rerun window, and result publication time will be announced with final deadlines.
Where authoritative changes will appear
Any change to a milestone, interface, evaluator, dataset, or rule will be versioned and announced through the official competition website and the support channel selected by the organizers.