[challenge]: The problem factory — a harness that generates, solves, and publishes its own autoresearch problems

Released by

Jin-Guo Liu

Contact email

jinguoliu@hkust-gz.edu.cn

Method

Other

Challenge issue

Background

The five companion autoresearch challenges (#124, #125, #126, #127, #128) were curated by an AI agent with a human in the loop, against two explicit selection criteria: (i) a well-posed theorem proof or optimization with a clear, ungameable bar; (ii) passing the bar delivers a meaningful publication. This meta-challenge asks whether that curation-and-solving loop can itself be automated. The claim under test: problem selection — not problem solving — is the real bottleneck of autonomous research. Systems like FunSearch, AlphaEvolve, and the AI Scientist have shown that agents can search heuristic spaces and draft papers; none has demonstrated an end-to-end loop that finds its own publishable problems, solves them against pre-registered machine-checkable gates, and survives human peer review.

Research objective

Build a harness with two components and run it to completion:

1. A problem generator with an executable quality rubric. The generator mines the literature for SOTA tables, open certified-bound gaps, and disputed results, and emits candidate autoresearch problems. Every candidate must ship its verification gate as runnable code — an SDP/dual-feasibility certificate checker, an interval-arithmetic inequality, deterministic cost arithmetic, or a fresh-sample re-evaluation protocol. A candidate that cannot express its gate as code is automatically rejected. Human involvement in generation is confined to a declared gatekeeping role, and every rejection (automatic or human) is logged.
2. A solver with an accumulating heuristics library. The solver runs autoresearch on accepted problems. After each attempt — success or failure — it distills what worked into persistent, reusable heuristics (skills, strategies, tool recipes), so problem N+1 starts from a richer library than problem N. The library and its growth curve are a reported output of the challenge, not an implementation detail.

Goal: generate 5 new challenge problems in quantum many-body simulation, solve them, and publish the solutions — where "publish" means passing refereed human peer review.

Success tiers

• Tier 1 — Generate. 5 new problems, each with an executable gate, accepted into this challenge catalog by human review.
• Tier 2 — Solve. All 5 gates pass on fresh evaluation; the heuristics library is published.
• Tier 3 — Publish (final bar). Solutions are written up as manuscripts and pass refereed peer review (journal or refereed venue; an arXiv posting alone does not count).

Tiers 1 and 2 make the challenge scoreable within the school's timeline; Tier 3 defines full success and is expected to outlive it.

Verification plan

What keeps the meta level ungameable:

• Calibration gate. Before generating new problems, the generator must re-derive problems of the same quality class as #124–#128 from the open literature, without access to the originals. If the rubric cannot reconstruct the hand-curated set, it is not trusted on new problems.
• Novelty check. Generated problems must not duplicate the existing catalog or any published problem list. The provenance of every candidate (papers mined, numbers extracted) is traced and published.
• Reproducible generation. The full generation run — prompts, transcripts, rejection log — is published alongside the accepted problems. Curation smuggled in by undeclared human editing is detectable by anyone reading the log.
• Pre-registered solving gate. Each problem's gate is frozen at generation time. A problem counts as solved only when its own shipped gate passes on fresh evaluation (fresh samples, independent chains, or exact certificate checking, as the gate specifies).
• External final gate. Peer review is performed by humans with no stake in the harness, so Tier 3 cannot be gamed from inside the system.

Why this may lead to research output

Twice over. Each solved problem is, by construction, its own publication — the rubric only admits problems whose bar-passing is publishable (that is what the rubric is). And the harness itself is a second, meta-level result: a documented, reproducible answer to "can an agent pick its own research problems well?", with the calibration test, rejection logs, and heuristics-library growth curve as evidence. Even partial failure is informative — a generator that produces only gameable or unpublishable problems, with logs showing why, is a real finding about the limits of autonomous research.

References

• Companion hand-curated problems: #124, #125, #126, #127, #128 (the calibration set).
• B. Romera-Paredes et al., "Mathematical discoveries from program search with large language models" (FunSearch), Nature 625, 468 (2024).
• A. Novikov et al., "AlphaEvolve: a coding agent for scientific and algorithmic discovery", arXiv:2506.13131 (2025).
• C. Lu et al., "The AI Scientist: towards fully automated open-ended scientific discovery", arXiv:2408.06292 (2024).
• Selection criteria inherited from the companion set: a well-posed bar that cannot be gamed, whose passing is worth a paper.

No files, tests, or entry points are named. Start by inspecting the repository and reviewing companion issues #124–#128 as the calibration set, then define the generator, executable gates, solver, and heuristics library. Done means producing five accepted problems, passing all five fresh-evaluation gates, and documenting the generation logs, library growth, and publication outcome.