OWASP / OWASP/cve-lite-cli

perf(usage): stripComments makes import extraction ~11x slower

Open
#1,120 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement in-house
Dominant language
TypeScript
Stars
715
Forks
145
Avg merge
21h 39m
Merged PRs (30d)
66

Description

Note: this is an in-house item already being handled by the maintainer - not open for contribution. Filed for tracking only.

Measured while reviewing #1114, which adds a stripComments pass to scanAllImports so JSDoc and commented-out imports stop counting as usage. The pass is correct and worth keeping. It is also about 11x slower than what it replaces.

Measured

src/ plus tests/ in this repo, 271 files, 1.9 MB, averaged over 5 passes:

read only (I/O) :  3.2 ms/pass
old extract     :  1.2 ms/pass
new extract     : 13.0 ms/pass

So the scanning phase goes from roughly 4.4 ms to 16.2 ms, with the extraction step itself about 11x. At the 5,000-file cap that is roughly 240 ms against 22 ms.

Cause

stripComments builds its result with per-character out += ch over the whole file, materialising a second copy of every source file. The pre-filter in scanAllImports (content.includes("import") || "require" || "export") admits essentially every source file, so almost nothing is skipped.

Options

Accumulate chunks in an array and join("") at the end, or compute comment ranges once and filter regex matches by index rather than building a stripped copy at all. The second avoids the allocation entirely and is probably the better shape, since the caller only needs match positions.

Priority

Not urgent. Absolute numbers are small and correctness matters more than the milliseconds here. Filed because #837 (CLI performance audit) is open in P1 and this is a known, measured, cheap-to-fix regression that should be folded into that work rather than rediscovered later.

Should land after #1114 merges, since it modifies code that PR introduces.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the stripComments pass and its caller, scanAllImports, in the changes introduced by #1114. Reproduce the five-pass measurements on the 271-file src/ and tests/ set, then verify that commented imports remain excluded while the extraction regression is reduced. This item is already being handled by the maintainer and is intended to fold into the #837 performance audit.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
cli, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.