apex-dev-tools / apex-dev-tools/apex-log-parser

✨ feat: Improve parse performance and memory, benchmark-led

Đang mở
#37 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
enhancement
Ngôn ngữ chính
TypeScript
Star
2
Fork
0
Merge trung bình
5 giờ 51 phút
Pull request đã merge (30 ngày)
32

Mô tả

### Problem

Parse cost is felt directly by both consumers, and both have had to build around it:

- The analyzer parses on the main thread of the webview, so a large log blocks the UI.
- MCP holds a single-slot fingerprinted cache with a five-minute idle eviction to avoid re-parsing,
and documents the memory cost in `src/tools/apexLogSource.ts:67-73`: the tree takes *"four to five
times the size of the file … a 200 MB log holds about a gigabyte"*. It also reads the whole file
into a string before parsing (`handle.readFile("utf-8").then(parse)`, line 126).

Both numbers — time and retained memory — need to come down.

### Proposed solution

#### Ground rule

**A benchmark harness lands first, and no change merges without a number.** Every item below is a
candidate, not a decision. If a change does not measurably improve time or retained memory on real
logs, it does not go in.

The harness needs a fixed set of real logs spanning sizes (small, ~20 MB, and one very large),
reporting parse wall time, peak resident memory and retained tree size, stable enough to compare two
commits.

This issue is deliberately one issue; split it once the harness shows where the cost is.

#### Candidates

**Passes over the tree.** Rollups run as separate post-passes today — `flattenByDepth` plus
`aggregateTotals` bottom-up, then `applyFlowDbResiduals`, then `resolveIssueEndTimes`. Compute totals
when a node closes, during `parseTree`, and fold `resolveIssueEndTimes` into the same pass.

**Per-event allocation.** Each event is a class instance with a `children` array, a split `parts`
array and several strings retained from the source. Most events have no children. Look at lazy child
arrays, interning repeated strings (type, namespace), and avoiding the intermediate `parts` array.
This is where the 4–5× file-size figure comes from.

**Non-blocking parse.** A `parseAsync` that yields periodically, so the analyzer's webview stays
responsive. Cheaper than moving to a worker, and it needs a progress signal to be useful.

**Streaming source.** Accept an `AsyncIterable` or a Node stream, so a 200 MB log is never
held as one string. Pairs with `parseAsync`.

**Lazy or columnar spike.** A structure-of-arrays event store, with the object view materialised on
demand. Highest risk, potentially the largest win on memory. Spike and measure before committing —
it changes the public model, so it needs a decision, not a merge.

#### Acceptance

- The harness is in the repo and runnable in CI.
- Every merged change cites a before and after number from it.
- Anything measured and rejected is recorded, so it is not retried blind.

### Alternatives considered

Weighed inside **Candidates** above; the ground rule is that none is chosen without a number.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.