block / block/trailblaze

NetworkLogSource: lazy/streaming load for sessions exceeding MAX_DISPLAY_LINES

Open
#126 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Kotlin
Stars
310
Forks
28
Avg merge
7h 45m
Merged PRs (30d)
8

Description

## Context

[`NetworkLogSource.parse`](https://github.com/block/trailblaze/blob/main/opensource/trailblaze-ui/src/commonMain/kotlin/xyz/block/trailblaze/ui/tabs/session/NetworkLogSource.kt) currently caps rendered events at `MAX_DISPLAY_LINES = 5_000`, keeping only the most recent N via `takeLast`. For typical web sessions (a few hundred to low thousands of events) this is invisible; for a 30-minute test on a chatty SPA hitting tens of thousands of requests, the panel only shows the last 5_000 and the rest are silently dropped from the view.

The cap was kept deliberately for the initial wiring; this issue tracks lifting it.

## Why a hard cap exists

Two reasons:
1. `LazyColumn` stays responsive only up to a point; rendering 50k JSON-derived rows starts to stutter on scrub.
2. `parse()` materializes the full filtered list before slicing (`lineSequence().filter { … }.toList().takeLast(N)`) — the upper bound on memory cost is set by the cap, not by file size.

## Strategies to evaluate

**A) Streaming parse with a ring buffer.**
Replace `lineSequence().filter{}.toList().takeLast(N)` with a streaming pass that maintains an `ArrayDeque` of size N. O(1) memory per line during parse instead of O(file size). Doesn't change the cap, just stops doubling memory transiently. Cheapest win; ~5 LOC.

**B) Range-based windowed parse.**
Parse only events whose `timestampMs` falls within the user's currently scrubbed window (plus a buffer on either side). Re-parse on scrub. Requires keeping the file content in memory but materializing only the visible slice. Cap effectively becomes "N events around the cursor" rather than "last N events ever." Best UX for long sessions.

**C) Pre-built index file.**
At capture-write time (or first-load time), build a sidecar index of `(byteOffset, timestampMs)` per event. Range queries become disk-seek + parse-only-visible-bytes. Rules out re-parsing the whole file on every scrub. More machinery; pairs naturally with future response-body capture work.

**D) Two-pane: counter header + paginated detail.**
The header always shows the true total ("12,432 events captured"). The list shows a window with paging controls. No data is hidden, but interaction shifts from "scroll forever" to "page through ranges." Low-effort, high-clarity.

## Recommendation

Start with **A** as a free win (no API change, no behavior change for sessions under the cap), then evaluate **B** as a v2 that genuinely supports long sessions. **C** depends on broader on-disk schema changes. **D** is a UX call — nice to have alongside any of the above.

## Estimated diff

A: ~10 LOC. B: ~80 LOC. C: ~200 LOC + on-disk schema. D: ~50 LOC of Compose.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.