apex-dev-tools / apex-dev-tools/apex-log-parser

✨ feat: Improve parse performance and memory, benchmark-led

Ouverte
#37 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
enhancement
Langage dominant
TypeScript
Étoiles
2
Forks
0
Merge moyen
5 h 51 min
PR mergées (30 j)
32

Description

### Problem

Parse cost is felt directly by both consumers, and both have had to build around it:

- The analyzer parses on the main thread of the webview, so a large log blocks the UI.
- MCP holds a single-slot fingerprinted cache with a five-minute idle eviction to avoid re-parsing,
and documents the memory cost in `src/tools/apexLogSource.ts:67-73`: the tree takes *"four to five
times the size of the file … a 200 MB log holds about a gigabyte"*. It also reads the whole file
into a string before parsing (`handle.readFile("utf-8").then(parse)`, line 126).

Both numbers — time and retained memory — need to come down.

### Proposed solution

#### Ground rule

**A benchmark harness lands first, and no change merges without a number.** Every item below is a
candidate, not a decision. If a change does not measurably improve time or retained memory on real
logs, it does not go in.

The harness needs a fixed set of real logs spanning sizes (small, ~20 MB, and one very large),
reporting parse wall time, peak resident memory and retained tree size, stable enough to compare two
commits.

This issue is deliberately one issue; split it once the harness shows where the cost is.

#### Candidates

**Passes over the tree.** Rollups run as separate post-passes today — `flattenByDepth` plus
`aggregateTotals` bottom-up, then `applyFlowDbResiduals`, then `resolveIssueEndTimes`. Compute totals
when a node closes, during `parseTree`, and fold `resolveIssueEndTimes` into the same pass.

**Per-event allocation.** Each event is a class instance with a `children` array, a split `parts`
array and several strings retained from the source. Most events have no children. Look at lazy child
arrays, interning repeated strings (type, namespace), and avoiding the intermediate `parts` array.
This is where the 4–5× file-size figure comes from.

**Non-blocking parse.** A `parseAsync` that yields periodically, so the analyzer's webview stays
responsive. Cheaper than moving to a worker, and it needs a progress signal to be useful.

**Streaming source.** Accept an `AsyncIterable` or a Node stream, so a 200 MB log is never
held as one string. Pairs with `parseAsync`.

**Lazy or columnar spike.** A structure-of-arrays event store, with the object view materialised on
demand. Highest risk, potentially the largest win on memory. Spike and measure before committing —
it changes the public model, so it needs a decision, not a merge.

#### Acceptance

- The harness is in the repo and runnable in CI.
- Every merged change cites a before and after number from it.
- Anything measured and rejected is recorded, so it is not retried blind.

### Alternatives considered

Weighed inside **Candidates** above; the ground rule is that none is chosen without a number.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.