google-deepmind / google-deepmind/formal-conjectures
Feature: Difficulty rating system and ATP benchmarking attributes (@[difficulty])
- Dominant language
- Lean
- Stars
- 1.3k
- Forks
- 485
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 327
Description
I’d like to propose adding a metadata layer to the conjectures via a new @[difficulty] attribute. While the repository has excellent categorization, there isn't currently a standardized way to track the "hardness" of a problem—either for human formalizers or for Automated Theorem Provers (ATP).
The Proposal I suggest implementing an attribute system that allows contributors to tag conjectures with:
Complexity Tier (1-5): A scale from "Introductory/Trivial" to "Grand Challenge/Open Problem."
ATP Status: Metadata to track if the conjecture has been solved by systems like AlphaProof, Aesop, or Hammer, including the "search budget" or tactics used.
Formalization Effort: A metric for the complexity of the Lean statement itself (e.g., number of dependencies or definitions involved).
Example Usage:
Lean
@[difficulty { tier := 3, atp := "AlphaProof", formalization := "medium" }]
conjecture riemann_hypothesis : ...
Why this is useful for the project:
Curriculum Learning: It enables the creation of structured datasets for training AI models on a gradient of difficulty.
Gap Analysis: Helps identify which mathematical areas (via AMS codes) are currently most resistant to automation.
Benchmarking: Provides a native way to treat the repo as a living leaderboard for new formalization agents and tactic-suggestion models.
Implementation Plan: I can extend the existing attribute logic in Util/ to support these fields. This would allow external scripts to easily parse .lean files to generate progress reports or filtered datasets.
I'm happy to put together a PR for the attribute definitions if this aligns with the current roadmap for the repository.
Contributor guide
Research direction
Start by reviewing the existing attribute logic in Util/ and representative .lean conjecture declarations. Clarify the project roadmap and the intended schema for difficulty tiers, ATP status, and formalization effort before implementing anything; done means the agreed metadata can be applied to conjectures and parsed by external scripts.
Written by the indexing model from the issue text.
Assessment
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100