jonhoo / jonhoo/inferno

Flamegraph generation memory usage is quite high for large inputs

Open
#201 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2.2k
Forks
160
PR merge metrics
No merged PRs in 30d

Description

Memory usage for `inferno-flamegraph`, and the equivalent Rust API usage, is proportional to input file size. A 44MB file results in 60MB memory usage for me, a 3KB input file results in 3MB (presumably the minimum).

I discovered this when processing a 440MB file, which resulted in hundreds of MB RAM usage, which is embarrassing when one is implementing a memory profiler :grin: So now I'm prefiltering out tiny irrelevant frames, which is why it's 44MB and not 440MB. Still, less memory usage would be nice.

Now, the _output_ file is typically more like a 1 megabyte or less, because all those repeating frames in the input file get combined into a graph in the output. So it ought to be possible to reduce memory usage quite a lot in the internal representation as well.

My completely unverified guess as to the problem: my input files have quite long strings for frame names, and multiple copies of each string are being stored in memory when the data structures are built up. If this is the case, usage of a string interner in the right place might be quite helpful, and potentially even speed up runtime because more data would fit in the CPU memory caches.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the inferno-flamegraph input-processing path and the equivalent Rust API entry point, then measure memory use on the reported 44MB and 440MB inputs. Determine whether repeated frame-name storage dominates the representation; done means substantially lower peak memory with equivalent flamegraph output and coverage for the large-input case.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.