leanprover / leanprover/lean-eval
Post-overhaul UX, performance, and cleanup triage
Nobody has claimed this yet.
- Dominant language
- Lean
- Stars
- 46
- Forks
- 39
- Avg merge
- 54m
- Merged PRs (30d)
- 80
Description
This issue collects deliberately deferred work discovered during the lifecycle-overhaul final audit. None of it is required for the initial release unless separately promoted into the authoritative completion plan.
Submission experience
- Add a submit call-to-action on problem pages and a searchable problem selector.
- Accept convenient repository URLs/refs, resolve them to immutable commits, and ask the submitter to confirm the resolved commit.
- Add optional friendly metadata fields and a preview of the public submission/result presentation.
- Add a durable receipt/status page with useful progress information.
- Replace the current inferred OAuth state with an explicit server-confirmed signed-in state.
- Guide users through missing GitHub App access; reconsider consolidating the two app installations.
- Improve form styling and human-readable display of submission metadata.
Reliability and operations
- Display a stale-data warning on the public site and alert operators when publication stops advancing.
- Profile and reduce evaluation latency, including cache behavior, nanoda runtime, the six-hour ceiling, and memory/OOM behavior. Keep the existing generator, latency, nanoda, and OOM issues linked to this work rather than duplicating them.
- Reduce contention in State result callbacks.
- Profile and reduce the roughly 20-minute leaderboard Pages build so small copy and projection fixes do not dominate deployment wall-clock.
- Make cross-repository State contract rollouts avoid a production-disabled gap: validate the candidate contract before disabling the current Worker or restore the prior healthy version on pre-finalization failure.
Post-transition cleanup
- Delete one-shot migration machinery after the final issue-intake delta is complete.
- Simplify release automation after the post-embargo production canary.
- Split the large submission workflow, Worker application, and GitHub-backed state implementation into smaller maintainable units.
- Eventually delete disabled publication opt-out/revocation and model-consolidation code paths.
Issue hygiene after launch
- Close or reclassify stale/fixed reports, including lean-eval #511, lean-eval-leaderboard #53 and #64, the FC-related lean-eval #533, model-consolidation lean-eval-leaderboard #83, and orphaned lean-eval-submissions #585 and #1449.
- Link the existing generator, latency, nanoda, and OOM issues here once triage begins.
Calendar-bound cutoff, retirement, and canary work stays in the overhaul execution runbook and its existing PRs; it is intentionally not duplicated here.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
This is a cross-cutting triage list rather than a self-contained change. Start by comparing its deferred items with the authoritative completion plan, existing generator, latency, nanoda, and OOM issues, and the post-overhaul runbook. It is done when each item is split or linked to a scoped issue, with stale work reclassified and calendar-bound work left in the runbook.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, devops, full-stack
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100