haskell / haskell/ThreadScope

Enhance handling of large eventlogs

Open
#29 5 comments 0 reactions 0 assignees View on GitHub
@major enhancement
Dominant language
Haskell
Stars
173
Forks
37
PR merge metrics
No merged PRs in 30d

Description

**Original reporter**: _jan.stolarek@_

When I enable detailed spark logging via -lf flag I end up with huge eventlog files (130MB). Attempting to load these into ThreadScope practically kills my OS - memory runs out, swapping begins and I am forced to kill TS (which takes some time before the OS actually responds and kills the process). This makes -lf flag useless for my program and I think this might not be uncommon situation. It would be good if TS supported some sort of lazy loading of big eventlogs, so users could at least view parts of the log.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue concerns ThreadScope loading large eventlog files produced with the -lf flag, but it does not identify the relevant files, tests, or loading entry point. Start by tracing how ThreadScope opens and retains eventlogs, then define how users should view part of a log without exhausting memory. Done means a 130MB eventlog can be opened and inspected without the current memory failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
desktop, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.