haskell / haskell/ghc-events

Bounded memory sorting

Open
#32 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
37
Forks
35
PR merge metrics
No merged PRs in 30d

Description

While writing an analysis tool, the need for sorting the events in an eventlog file arose. The problem with the existing `GHC.RTS.Events.sortEvents` is that it is not memory bounded, leading to heap exhaustion on large inputs.

How about we provide a memory bounded version?

I ended up writing something similar to [this](https://gist.github.com/facundominguez/b257a6b1d79533131917066716cdfb13). It splits events per capability to different temporary files, and then merges the events back into a single list.

I could submit a PR if it looks useful to others.

This version is a bit wasteful in that it has to deserialize and serialize all the events, where for the sake of splitting the events, only the capability number and the encoded form of the event would be needed.

Another odd aspect is that I couldn't make it stream when using Data.ByteString.Builder to write the events to the temporary files. Heap would be exhausted the same.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with GHC.RTS.Events.sortEvents and the linked gist, then review the existing eventlog parsing and serialization entry points. A complete change would provide a memory-bounded sorting variant that handles large inputs without heap exhaustion, while resolving the temporary-file and streaming concerns described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
data, performance, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.