Flamegraph generation memory usage is quite high for large inputs
- Dominant language
- Rust
- Stars
- 2.2k
- Forks
- 160
- PR merge metrics
- No merged PRs in 30d
Description
Memory usage for `inferno-flamegraph`, and the equivalent Rust API usage, is proportional to input file size. A 44MB file results in 60MB memory usage for me, a 3KB input file results in 3MB (presumably the minimum).
I discovered this when processing a 440MB file, which resulted in hundreds of MB RAM usage, which is embarrassing when one is implementing a memory profiler :grin: So now I'm prefiltering out tiny irrelevant frames, which is why it's 44MB and not 440MB. Still, less memory usage would be nice.
Now, the _output_ file is typically more like a 1 megabyte or less, because all those repeating frames in the input file get combined into a graph in the output. So it ought to be possible to reduce memory usage quite a lot in the internal representation as well.
My completely unverified guess as to the problem: my input files have quite long strings for frame names, and multiple copies of each string are being stored in memory when the data structures are built up. If this is the case, usage of a string interner in the right place might be quite helpful, and potentially even speed up runtime because more data would fit in the CPU memory caches.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the inferno-flamegraph input-processing path and the equivalent Rust API entry point, then measure memory use on the reported 44MB and 440MB inputs. Determine whether repeated frame-name storage dominates the representation; done means substantially lower peak memory with equivalent flamegraph output and coverage for the large-input case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100