google-deepmind / google-deepmind/formal-conjectures

Feature: Difficulty rating system and ATP benchmarking attributes (@[difficulty])

Open
#1,608 6 comments 1 reaction 0 assignees View on GitHub
attributes enhancement
Dominant language
Lean
Stars
1.3k
Forks
485
Avg merge
1d 20h
Merged PRs (30d)
327

Description

I’d like to propose adding a metadata layer to the conjectures via a new @[difficulty] attribute. While the repository has excellent categorization, there isn't currently a standardized way to track the "hardness" of a problem—either for human formalizers or for Automated Theorem Provers (ATP).

The Proposal I suggest implementing an attribute system that allows contributors to tag conjectures with:

Complexity Tier (1-5): A scale from "Introductory/Trivial" to "Grand Challenge/Open Problem."

ATP Status: Metadata to track if the conjecture has been solved by systems like AlphaProof, Aesop, or Hammer, including the "search budget" or tactics used.

Formalization Effort: A metric for the complexity of the Lean statement itself (e.g., number of dependencies or definitions involved).

Example Usage:

Lean

@[difficulty { tier := 3, atp := "AlphaProof", formalization := "medium" }]
conjecture riemann_hypothesis : ...
Why this is useful for the project:

Curriculum Learning: It enables the creation of structured datasets for training AI models on a gradient of difficulty.

Gap Analysis: Helps identify which mathematical areas (via AMS codes) are currently most resistant to automation.

Benchmarking: Provides a native way to treat the repo as a living leaderboard for new formalization agents and tactic-suggestion models.

Implementation Plan: I can extend the existing attribute logic in Util/ to support these fields. This would allow external scripts to easily parse .lean files to generate progress reports or filtered datasets.

I'm happy to put together a PR for the attribute definitions if this aligns with the current roadmap for the repository.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing attribute logic in Util/ and representative .lean conjecture declarations. Clarify the project roadmap and the intended schema for difficulty tiers, ATP status, and formalization effort before implementing anything; done means the agreed metadata can be applied to conjectures and parsed by external scripts.

Written by the indexing model from the issue text.

Assessment

Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.