Add a way to generate flagemegraphs for files that are too big to fit in RAM
- Dominant language
- Rust
- Stars
- 2.2k
- Forks
- 160
- PR merge metrics
- No merged PRs in 30d
Description
I was recently trying to generate a flamegraph for a `nix` evaluation and the generated `.folded` file is just *too* big:
```console
$ ls -lh funktion-calls.folded
-rw-r--r-- 1 jalil users 127G Apr 13 10:27 funktion-calls.folded
$ inferno-flamegraph ./funktion-calls.folded
Error: Io(Kind(OutOfMemory))
```
This is better behavior than the `perl` impl that slowly uses more memory and crashes my computer, but I would like to be able to produce a flamegraph from it.
Looking at the code here is when the error happens:
https://github.com/jonhoo/inferno/blob/5ee7d6bb68ac9627010ef468c81f4f264cd0d399/src/flamegraph/mod.rs#L830-L841
Reading the whole file into memory is impossible, but looking further we could get up to here without reading the whole file: https://github.com/jonhoo/inferno/blob/5ee7d6bb68ac9627010ef468c81f4f264cd0d399/src/flamegraph/mod.rs#L446-L449
The other branches evaluate the whole iterator. The problem then becomes the `merge::frames` function which returns a `Vec<_>` (AFAICT this holds references to the file memory and thus needs to read it all).
I did not look too closely into the `merge::frames` implementation once I figured out it was probably not possible to make it lazy without a major rewrite.
I would like to know if there are any big blockers for this, if not, then I would like to try tackling it.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/flamegraph/mod.rs at the referenced handling around lines 830-841 and 446-449, then inspect merge::frames and its Vec return value. Determine whether oversized .folded files can be processed without reading the entire file into RAM; done means inferno-flamegraph produces a flamegraph from such input without an out-of-memory failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100