Memory leak building under WSL
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 816
- Forks
- 88
- PR merge metrics
- No merged PRs in 30d
Description
Describe the bug
Trying to build a large repo on WSL2 (Ubuntu 22.04.4 LTS). When I invoke lage, the process seems to grow monotonically and slow down until one of the build threads runs out of memory in the JS heap.
To Reproduce
This occurs in a private repo within Microsoft that I can't share here. Please contact me internally and I can help you with the repro.
- Clone the repo in WSL2,
yarn,yarn build
The build will start off transpiling packages very fast, but then the memory usage of the top level task will start growing. As it does, the overall build slows down.
Expected behavior
The build should complete.
Environment/Troubleshooting
I am running lage through node through yarn in our repo. The build environment is WSL2 running Ubuntu 22.04.4 LTS. I do not see this issue when I'm running the build inside of Windows
I hacked our internal lage tasks to dump a v8 Heap Snapshot at the time of the transpile starting. And there was definitely a leak visible between invocations of the next thread. Many of these appeared to be directly related to <graceful-fs.queue>.
I built my own local version of lage, replacing graceful-fs with stock fs. This reudce the rate of growth, but there was still a slowdown and eventual crash.
At which point I moved my instrumentation for Heap Snapshots to WorkerRunner.run. This slowed down the leak rate as well. I believe this was due to the v8.writeHeapSnapshot funciton invoking GC. However, this dump kept taking longer, so the build was still slow.
I then changed this code to a direct call to just invoke the GC before every run:
This highly reduced the memory growth, and the build was about to complete in 7m34s, which is faster than building the same repo on Windows was.
So I'm not sure if there's an actual leak or just a need for some idle time for the GC to run. There definitely appears to be an issue with graceful-fs under WSL2 & Ubuntu 22.04.4 in lage.
Here's the output form "top" while the process was running. This is a 16-core 64GB machine, so I had 16 lage workers running with each limited to 2GB of JS heap.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The report points to WorkerRunner.run and graceful-fs while running lage through Node on WSL2 and Ubuntu 22.04.4. Start by reproducing the yarn and yarn build workflow, then compare heap growth with and without the reported GC call; done means the build completes without unbounded memory growth or slowdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript, ubuntu
- Domain
- build-system, operating-systems, tooling
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100