bloomberg / bloomberg/memray

Implement heatmap plot of allocation sizes over time

Open
#220 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
15.2k
Forks
461
Avg merge
5d 3h
Merged PRs (30d)
10

Description

Following [Brendan Gregg's webpage on performance visualizations](https://www.brendangregg.com/heatmaps.html) we can try to implement a heatmap plot of memory allocation sizes over time. Here is an example of such heatmap (ignore the title as the heatmap is generated by reusing [Brendan Gregg's utilities](https://github.com/brendangregg/HeatMap)):

![heatmap](https://user-images.githubusercontent.com/11718525/192782997-c0170d25-eee2-49a3-9233-cb61f28d3241.svg)

The idea is that every vertical slice is a histogram of allocation sizes in a bucket that spans some time interval (normally the same time interval as we are using to take RSS snapshots). The plot shows therefore a collection of histograms over time. Check the [docs](https://www.brendangregg.com/heatmaps.html) for more information on how to read these.

Some points that @godlygeek raised over this default implementation:

> the fact that it only shows allocations and not deallocations makes it easy to be misled by it, too, I think - since a straight line across the chart showing all the allocations are the same size wouldn't tell you anything about whether more memory is being used as time goes on. I'd almost want to see heap size plotted over top of it...

He then proposed the following:

> What if we took the heat map idea, but made an allocation keep counting in every time interval until it is deallocated? So instead of a heat map of allocation sizes over time, it's a chart of allocated bytes over time broken down into size bands?
The changes is that instead of an allocation counting only for a single (x, y) cell on that chart, it counts for one x, multiple y - it keeps counting for every y from when it was allocated until it is deallocated
In this way, each row in the chart corresponds to some size range of allocations. Within each row, from left to right, we see every allocation that has occurred, and when it is freed. The more overlapping allocations there are at some point in time, the more saturated the colours on the chart are. This should basically just be taking the existing code, and making it so that
> * Each column starts as a copy of the previous column, and
> * Instead of ignoring deallocations, they remove a value from that column's data set (or, better yet, from the next one) (edited)

Following this idea we can modify the initial proposal to make a "heap size distribution plot over time" which is likely more useful than allocations over time.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing allocation heatmap implementation referenced in the issue and the linked Brendan Gregg heatmap documentation. Define the heap size distribution plot so columns carry allocations forward until deallocation, then verify that the visualization represents allocation size bands and overlapping live allocations over time.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-visualization
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.