python / python/mypy

mypy daemon consumes an increasing amount of memory every time `run` or `recheck` is invoked

Open
#19,644 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-daemon
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

Bug Report

The amount of memory that the daemon process consumes doesn't drop when typing errors are fixed; the memory consumption appears to stay the same if there are no errors, and when more errors appear, the memory consumption just continues increasing.

This is problematic when running the daemon on initially untyped code bases with large amounts of typing errors (as usually done in code editors integrating with a type-checker), as dmypy becomes sluggish very quickly (because the same existing errors are repeatedly reported).

To Reproduce & Actual Behaviour

  1. Repeat this line 2000 times in a file project/test/__init__.py,

    aaaaaaaaaaaaaaaaaaaaaaaa
    

    and create the following mypy configuration file at project/mypy.ini

    [mypy]
    files = test/
    
  2. Navigate to project/, then run dmypy run > errors.txt multiple times without editing project/test/__init__.py; watch the memory consumption grow for the dmypy process.

  3. Delete all code in project/test/__init__.py, then run dmypy run again multiple times. The memory consumption does not decrease.

Expected Behavior

I expect the memory consumption to be proportional to the number of errors, AST nodes, and size of a cache diff since the last time the checked code files have changed. If there's no code changes, it's surprising to me that memory consumption would grow for just repeated reports of the same errors.

Your Environment

  • Mypy version used: 1.17.1
  • Mypy command-line flags: None
  • Mypy configuration options from mypy.ini (and other config files): See # project/mypy.ini
  • Operating system: Ubuntu (Pop!_OS) 22.04 LTS
  • Python version used: 3.11

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue from project/test/init.py using project/mypy.ini, then invoke dmypy run and recheck repeatedly while observing daemon memory. Compare behavior with the repeated errors removed; done means memory no longer grows from repeated reports and decreases when errors and AST nodes are removed, with regression coverage for the scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.