leanprover / leanprover/lean4

Lean server keeps consuming more memory as it reprocesses a large definition

Open
#6,753 8 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug P-medium
Dominant language
Lean
Stars
9.2k
Forks
990
Avg merge
1d 17h
Merged PRs (30d)
175

Description

Prerequisites

Please put an X between the brackets as you perform the following steps:

Description

When interacting with a Lean server through VSCode and editing mid-file, the server often reprocess the definition later in the file. When a definition later in the file has many arguments with complex types, the server process consumes additional 1-2GB of RAM as it reprocess the definition, and it doesn't free the memory until my WSL environment crashes.

When setting LEAN_NUM_THREADS=1, the server still consumes additional memory, but the total memory usage of the server process will be capped at some point.

Context

This was originally reported at Zulip. The problematic definition is a custom induction principle that has 9 branches, and each branch takes 5-10 arguments.

Steps to Reproduce

I tried hard to minimize the issue, but it was difficult since commenting out parts of the definition reduces the additional memory usage significantly (often by an order of magnitude). At least the following code reproduces the issue without any dependencies in my environment.

Reproduction code
import Std.Data.HashSet

set_option autoImplicit false

open String (Pos Iterator)
open Std (HashSet)

#eval Lean.versionString -- "4.16.0-nightly-2025-01-23"

inductive Node where
  | done
  | fail
  | epsilon (next : Nat)
  | char (c : Char) (next : Nat)
  | split (next₁ next₂ : Nat)
  | save (offset : Nat) (next : Nat)
  | sparse (cs : Char) (next : Nat)
deriving Repr

structure NFA where
  nodes : Array Node
  start : Nat
deriving Repr

def get (nfa : NFA) (i : Nat) (h : i < nfa.nodes.size) : Node :=
  nfa.nodes[i]

instance : GetElem NFA Nat Node (fun nfa i => i < nfa.nodes.size) where
  getElem nfa i h := get nfa i h

structure SearchState' (nfa : NFA) where
  states : HashSet (Fin nfa.nodes.size)
  updates : Vector (List (Nat × Pos)) nfa.nodes.size

abbrev εStack' (nfa : NFA) := List (List (Nat × Pos) × Fin nfa.nodes.size)

-- Run `top` and press Shift + M to see most memory-consuming processes.
-- Editing this comment will cause Lean server to re-check `εClosure'.induct'`,
-- leading to additional 1-2GB of memory consumption.
-- Don't forget monitoring the memory usage and restart the server before your system hangs with high memory pressure :)
theorem εClosure'.induct' (nfa : NFA) (it : Iterator)
  (motive : Option (List (Nat × Pos)) → SearchState' nfa → εStack' nfa → Prop)
  (base : ∀ matched next, motive matched next [])
  (visited :
    ∀ matched next update state stack',
    state ∈ next.states →
    motive matched next stack' →
    motive matched next ((update, state) :: stack'))
  (epsilon : ∀ matched next update state stack' state',
    state ∉ next.states → nfa[state] = .epsilon state' →
    let next' := ⟨next.states.insert state, next.updates⟩;
    motive matched next' ((update, state') :: stack') →
    motive matched next ((update, state) :: stack'))
  (split : ∀ matched next update state stack' state₁ state₂,
    state ∉ next.states → nfa[state] = .split state₁ state₂ →
    let next' := ⟨next.states.insert state, next.updates⟩;
    motive matched next' ((update, state₁) :: (update, state₂) :: stack') →
    motive matched next ((update, state) :: stack'))
  (save : ∀ matched next update state stack' offset state',
    state ∉ next.states → nfa[state] = .save offset state' →
    let next' := ⟨next.states.insert state, next.updates⟩;
    motive matched next' ((update ++ [(offset, it.pos)], state') :: stack') →
    motive matched next ((update, state) :: stack'))
  (done : ∀ matched next update state stack',
    state ∉ next.states → nfa[state] = .done →
    let next' := ⟨next.states.insert state, next.updates.set state update⟩;
    motive (matched <|> .some update) next' stack' →
    motive matched next ((update, state) :: stack'))
  (char : ∀ matched next update state stack' c (state' : Fin nfa.nodes.size),
    state ∉ next.states → nfa[state] = .char c state' →
    let next' := ⟨next.states.insert state, next.updates.set state update⟩;
    motive matched next' stack' →
    motive matched next ((update, state) :: stack'))
  (sparse : ∀ matched next update state stack' cs (state' : Fin nfa.nodes.size),
    state ∉ next.states → nfa[state] = .sparse cs state' →
    let next' := ⟨next.states.insert state, next.updates.set state update⟩;
    motive matched next' stack' →
    motive matched next ((update, state) :: stack'))
  (fail : ∀ matched next update state stack',
    state ∉ next.states → nfa[state] = .fail →
    let next' := ⟨next.states.insert state, next.updates⟩;
    motive matched next' stack' →
    motive matched next ((update, state) :: stack')) :
  ∀ matched next stack, motive matched next stack := sorry
  1. Run top and press Shift + M to monitor the most memory-consuming processes
  2. Copy the reproduction code in VSCode and watch the memory usage
  • You can find the server process with ps aux | grep lean
  1. Edit comments above εClosure'.induct' and watch Lean reprocess εClosure'.induct'
  2. Look at the memory usage

Expected behavior: [Clear and concise description of what you expect to happen]

Reprocessing εClosure'.induct' shouldn't increase the memory usage of the Lean server process significantly.

Actual behavior: [Clear and concise description of what actually happens]

Reprocessing εClosure'.induct' causes the Lean server process to consume additional 1-2GB of RAM.

Versions

Lean: "4.16.0-nightly-2025-01-23"
OS: Ubuntu 22.04.4 LTS in WSL2 (5.15.167.4-microsoft-standard-WSL2)

Additional Information
Impact

Add 👍 to issues you consider important. If others are impacted by this issue, please ask them to add 👍 to it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied reproduction in VSCode and monitor the Lean server with top or ps as described. Edit the comment above εClosure'.induct' to reproduce the reprocessing and compare memory usage. Done means repeated reprocessing no longer causes significant additional memory growth.

Written by the indexing model from the issue text.

Assessment

Domain
compilers, devtools
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.