apex-dev-tools / apex-dev-tools/apex-log-parser
✨ feat: Improve parse performance and memory, benchmark-led
- Langage dominant
- TypeScript
- Étoiles
- 2
- Forks
- 0
- Merge moyen
- 5 h 51 min
- PR mergées (30 j)
- 32
Description
### Problem
Parse cost is felt directly by both consumers, and both have had to build around it:
- The analyzer parses on the main thread of the webview, so a large log blocks the UI.
- MCP holds a single-slot fingerprinted cache with a five-minute idle eviction to avoid re-parsing,
and documents the memory cost in `src/tools/apexLogSource.ts:67-73`: the tree takes *"four to five
times the size of the file … a 200 MB log holds about a gigabyte"*. It also reads the whole file
into a string before parsing (`handle.readFile("utf-8").then(parse)`, line 126).
Both numbers — time and retained memory — need to come down.
### Proposed solution
#### Ground rule
**A benchmark harness lands first, and no change merges without a number.** Every item below is a
candidate, not a decision. If a change does not measurably improve time or retained memory on real
logs, it does not go in.
The harness needs a fixed set of real logs spanning sizes (small, ~20 MB, and one very large),
reporting parse wall time, peak resident memory and retained tree size, stable enough to compare two
commits.
This issue is deliberately one issue; split it once the harness shows where the cost is.
#### Candidates
**Passes over the tree.** Rollups run as separate post-passes today — `flattenByDepth` plus
`aggregateTotals` bottom-up, then `applyFlowDbResiduals`, then `resolveIssueEndTimes`. Compute totals
when a node closes, during `parseTree`, and fold `resolveIssueEndTimes` into the same pass.
**Per-event allocation.** Each event is a class instance with a `children` array, a split `parts`
array and several strings retained from the source. Most events have no children. Look at lazy child
arrays, interning repeated strings (type, namespace), and avoiding the intermediate `parts` array.
This is where the 4–5× file-size figure comes from.
**Non-blocking parse.** A `parseAsync` that yields periodically, so the analyzer's webview stays
responsive. Cheaper than moving to a worker, and it needs a progress signal to be useful.
**Streaming source.** Accept an `AsyncIterable` or a Node stream, so a 200 MB log is never
held as one string. Pairs with `parseAsync`.
**Lazy or columnar spike.** A structure-of-arrays event store, with the object view materialised on
demand. Highest risk, potentially the largest win on memory. Spike and measure before committing —
it changes the public model, so it needs a decision, not a merge.
#### Acceptance
- The harness is in the repo and runnable in CI.
- Every merged change cites a before and after number from it.
- Anything measured and rejected is recorded, so it is not retried blind.
### Alternatives considered
Weighed inside **Candidates** above; the ground rule is that none is chosen without a number.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.