felladrin / felladrin/MiniSearch
perf: keep long tasks off the main thread during search and generation
- Dominant language
- TypeScript
- Stars
- 587
- Forks
- 70
- Avg merge
- 1h 3m
- Merged PRs (30d)
- 175
Description
### Problem
There is no worker anywhere in `client/`, so tokenizing, markdown parsing, syntax highlighting and layout measurement all run on the same thread that handles scrolling, typing and the streaming answer.
Part of this is already handled: `updateResponse` is throttled to 12 updates per second (`client/modules/pubSub.ts:86`) and `ChatInterface` throttles the chat stream the same way. The rest is not.
- `getFormattedSearchResults()` calls `allocatePageExcerpts()`, which runs `gptTokenizer.encode()` over the full body of every page read by `/page-content` (`client/modules/textGenerationUtilities.ts:62`). It runs synchronously, right at the handoff from search to generation, over what can be several complete articles.
- `updateRollingSummary()` re-joins the kept parts and re-encodes the whole candidate string on every iteration (`client/modules/textGeneration.ts:206-212`), so the cost grows quadratically with the conversation length.
- `client/modules/shiki.ts:9` builds the highlighter with `Object.keys(bundledLanguages)` from `shiki/bundle/full`. The first code block in an answer pays for every language in the bundle instead of the one it needs.
- Every throttled frame re-parses the entire markdown string from scratch in `MarkdownRenderer.tsx`. The throttle keeps the frame rate at 12/s, but the work per frame keeps growing as the answer gets longer.
- `ExpandableLink` adds one unthrottled `resize` listener per citation link, and each listener reads `scrollWidth` and calls `setState` (`client/components/AiResponse/ExpandableLink.tsx:50-63`). It also animates `width`, which forces layout, instead of a property the compositor can handle alone.
I found all of these by reading the code, not by profiling, so the order above is a guess until someone measures it.
### Solution
Measure first, then take the heavy work off the render path: tokenization into a worker (or a single incremental count instead of the per-candidate loop), Shiki loading only the languages a response actually contains, block-level memoized markdown so a streamed frame re-renders only the block that changed, and one shared observer for the citation links.
### Acceptance criteria
- A performance profile of one search with AI response and page content enabled, recorded before and after, showing which long tasks were removed.
- Page content tokenization no longer blocks the main thread at the search-to-generation handoff.
- `updateRollingSummary()` encodes each part once instead of re-encoding the joined candidate on every iteration.
- Shiki loads only the languages present in the rendered answer.
- Streaming a long answer re-renders only the markdown block that changed.
- One resize observer for all citation links, and the expand animation uses `transform` and `opacity` only.
- An E2E check that fails if a task longer than 200 ms appears on the main thread while an answer streams.
### Implementation notes
Size: **L**. Each item is independent, so one PR per item.
- `client/modules/textGenerationUtilities.ts` - move the `allocatePageExcerpts()` tokenization into a worker, or budget by character count and tokenize only the trimmed result.
- `client/modules/textGeneration.ts` - keep a running token count in `updateRollingSummary()`.
- `client/modules/shiki.ts` - pass the languages found in the content, or switch to `shiki/bundle/web`.
- `client/components/AiResponse/MarkdownRenderer.tsx` - split the content into blocks and memoize each one.
- `client/components/AiResponse/ExpandableLink.tsx` - one shared `ResizeObserver`, and animate `transform` instead of `width`.
- `e2e/smoke.spec.ts` or a new spec - the long-task assertion, via `PerformanceObserver` on `longtask`.
References:
- [The Browser's Main Thread Is Expensive](https://kciter.so/posts/the-expensive-main-thread/en/)
Contributor guide
Research direction
Start with a performance profile of the search and streaming flow, then read the named entry points in client/modules/textGenerationUtilities.ts, client/modules/textGeneration.ts, client/modules/shiki.ts, MarkdownRenderer.tsx, and ExpandableLink.tsx. Treat each implementation note as an independent change and use e2e/smoke.spec.ts for the long-task check. Done means the listed main-thread, tokenization, highlighting, markdown, resize, animation, and streaming criteria are verified before and after profiling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- frontend, performance, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100