leanprover / leanprover/lean-eval

Post-overhaul UX, performance, and cleanup triage

Open
#634 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Lean
Stars
46
Forks
39
Avg merge
54m
Merged PRs (30d)
80

Description

This issue collects deliberately deferred work discovered during the lifecycle-overhaul final audit. None of it is required for the initial release unless separately promoted into the authoritative completion plan.

Submission experience

  • Add a submit call-to-action on problem pages and a searchable problem selector.
  • Accept convenient repository URLs/refs, resolve them to immutable commits, and ask the submitter to confirm the resolved commit.
  • Add optional friendly metadata fields and a preview of the public submission/result presentation.
  • Add a durable receipt/status page with useful progress information.
  • Replace the current inferred OAuth state with an explicit server-confirmed signed-in state.
  • Guide users through missing GitHub App access; reconsider consolidating the two app installations.
  • Improve form styling and human-readable display of submission metadata.

Reliability and operations

  • Display a stale-data warning on the public site and alert operators when publication stops advancing.
  • Profile and reduce evaluation latency, including cache behavior, nanoda runtime, the six-hour ceiling, and memory/OOM behavior. Keep the existing generator, latency, nanoda, and OOM issues linked to this work rather than duplicating them.
  • Reduce contention in State result callbacks.
  • Profile and reduce the roughly 20-minute leaderboard Pages build so small copy and projection fixes do not dominate deployment wall-clock.
  • Make cross-repository State contract rollouts avoid a production-disabled gap: validate the candidate contract before disabling the current Worker or restore the prior healthy version on pre-finalization failure.

Post-transition cleanup

  • Delete one-shot migration machinery after the final issue-intake delta is complete.
  • Simplify release automation after the post-embargo production canary.
  • Split the large submission workflow, Worker application, and GitHub-backed state implementation into smaller maintainable units.
  • Eventually delete disabled publication opt-out/revocation and model-consolidation code paths.

Issue hygiene after launch

  • Close or reclassify stale/fixed reports, including lean-eval #511, lean-eval-leaderboard #53 and #64, the FC-related lean-eval #533, model-consolidation lean-eval-leaderboard #83, and orphaned lean-eval-submissions #585 and #1449.
  • Link the existing generator, latency, nanoda, and OOM issues here once triage begins.

Calendar-bound cutoff, retirement, and canary work stays in the overhaul execution runbook and its existing PRs; it is intentionally not duplicated here.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

This is a cross-cutting triage list rather than a self-contained change. Start by comparing its deferred items with the authoritative completion plan, existing generator, latency, nanoda, and OOM issues, and the post-overhaul runbook. It is done when each item is split or linked to a scoped issue, with stale work reclassified and calendar-bound work left in the runbook.

Written by the indexing model from the issue text.

Assessment

Domain
backend, devops, full-stack
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.