jonhoo / jonhoo/inferno

Add a way to generate flagemegraphs for files that are too big to fit in RAM

Open
#342 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2.2k
Forks
160
PR merge metrics
No merged PRs in 30d

Description

I was recently trying to generate a flamegraph for a `nix` evaluation and the generated `.folded` file is just *too* big:

```console
$ ls -lh funktion-calls.folded
-rw-r--r-- 1 jalil users 127G Apr 13 10:27 funktion-calls.folded
$ inferno-flamegraph ./funktion-calls.folded
Error: Io(Kind(OutOfMemory))
```

This is better behavior than the `perl` impl that slowly uses more memory and crashes my computer, but I would like to be able to produce a flamegraph from it.

Looking at the code here is when the error happens:
https://github.com/jonhoo/inferno/blob/5ee7d6bb68ac9627010ef468c81f4f264cd0d399/src/flamegraph/mod.rs#L830-L841

Reading the whole file into memory is impossible, but looking further we could get up to here without reading the whole file: https://github.com/jonhoo/inferno/blob/5ee7d6bb68ac9627010ef468c81f4f264cd0d399/src/flamegraph/mod.rs#L446-L449

The other branches evaluate the whole iterator. The problem then becomes the `merge::frames` function which returns a `Vec<_>` (AFAICT this holds references to the file memory and thus needs to read it all).

I did not look too closely into the `merge::frames` implementation once I figured out it was probably not possible to make it lazy without a major rewrite.

I would like to know if there are any big blockers for this, if not, then I would like to try tackling it.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/flamegraph/mod.rs at the referenced handling around lines 830-841 and 446-449, then inspect merge::frames and its Vec return value. Determine whether oversized .folded files can be processed without reading the entire file into RAM; done means inferno-flamegraph produces a flamegraph from such input without an out-of-memory failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.