nodejs / nodejs/node

Running a process with DevTools debugger attached leaks memory

Open
#39,063 4 comments 13 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

inspector memory
Dominant language
JavaScript
Stars
122k
Forks
37.3k
Avg merge
4d 2h
Merged PRs (30d)
283

Description

While the Chromium DevTools debugger is attached, RSS memory usage of a Node process keeps increasing without being collected. Once the debugger is detached, the memory usage flattens, however the memory is never released. When the debugger is not attached to the process in its runtime, memory allocation occurs as expected. It is entirely possible that this might not be related to Node (v8, perhaps) - however, this issue might be a good place to start a discussion.

Inspecting htop on a Raspberry Pi 3B machine:

  • Process after running for ~24 hours without a debugger attached: stable at around ~180MB RSS:
    no debugger

  • The same process, after attaching the debugger for ~15 minutes (with no actual user debugging active nor any logs being displayed): RSS at around ~245MB and rising:
    debugger

  • After disconnecting the debugger (removing the connection, closing and re-opening the debugger window), RSS stabilizes at around ~260MB. After ~30 minutes of further activity, the RSS usage seems stable; however, the memory is not freed:
    after

After running valgrind for various amounts of time on a different Linux machine, there seems to be an excessive amount of v8_inspector::AsyncStackTrace::capture calls. Using the Disable async stack traces option in DevTools > Settings > Debugger seems to reduce the leak somewhat, however, this wasn't reproducible across machines. Furthermore, testing the same process on Windows 10 in the same environment leads to no memory leaks.

Attached are valgrind dumps for various tested nodejs versions. Note that each dump was created with the process running for differing amounts, hence the total memory leaked will not be as relevant.

valgrind.zip

What steps will reproduce the bug?
  1. Set up a Node process, possibly with asynchronous stack activity
  2. Attach a Chrome DevTools debugger instance
  3. Observe a memory leak as long as the debugger is attached
How often does it reproduce? Is there a required condition?
  • Only seems to reproduce on Linux, not Windows
  • Might be related to async stack traces
  • At first, seemed to be related to the mariadb NPM module - but upon further testing with the module inactive, the memory leak kept occuring
What is the expected behavior?

No memory leak should occur as the debugger is being used.

What do you see instead?

Depending on the process being run, upwards of hundreds of MB of RAM will leak in a couple of minutes as the debugger is attached.

Additional information

A similar type of memory leak seems to be present in versions prior to 14.0, however, in a much smaller scale, and occuring much more slowly. Reference: nodejs/node#28787, possibly nodejs/node#28420

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the leak using the listed steps on Linux, with Chrome DevTools attached, and compare runs with async stack traces enabled and disabled. Inspect the supplied Valgrind dumps and the excessive v8_inspector::AsyncStackTrace::capture activity; done means debugger use no longer causes sustained RSS growth or unreleased memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
performance, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.