felladrin / felladrin/MiniSearch

perf: keep long tasks off the main thread during search and generation

Open
#2,571 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
TypeScript
Stars
587
Forks
70
Avg merge
1h 3m
Merged PRs (30d)
175

Description

### Problem

There is no worker anywhere in `client/`, so tokenizing, markdown parsing, syntax highlighting and layout measurement all run on the same thread that handles scrolling, typing and the streaming answer.

Part of this is already handled: `updateResponse` is throttled to 12 updates per second (`client/modules/pubSub.ts:86`) and `ChatInterface` throttles the chat stream the same way. The rest is not.

- `getFormattedSearchResults()` calls `allocatePageExcerpts()`, which runs `gptTokenizer.encode()` over the full body of every page read by `/page-content` (`client/modules/textGenerationUtilities.ts:62`). It runs synchronously, right at the handoff from search to generation, over what can be several complete articles.
- `updateRollingSummary()` re-joins the kept parts and re-encodes the whole candidate string on every iteration (`client/modules/textGeneration.ts:206-212`), so the cost grows quadratically with the conversation length.
- `client/modules/shiki.ts:9` builds the highlighter with `Object.keys(bundledLanguages)` from `shiki/bundle/full`. The first code block in an answer pays for every language in the bundle instead of the one it needs.
- Every throttled frame re-parses the entire markdown string from scratch in `MarkdownRenderer.tsx`. The throttle keeps the frame rate at 12/s, but the work per frame keeps growing as the answer gets longer.
- `ExpandableLink` adds one unthrottled `resize` listener per citation link, and each listener reads `scrollWidth` and calls `setState` (`client/components/AiResponse/ExpandableLink.tsx:50-63`). It also animates `width`, which forces layout, instead of a property the compositor can handle alone.

I found all of these by reading the code, not by profiling, so the order above is a guess until someone measures it.

### Solution

Measure first, then take the heavy work off the render path: tokenization into a worker (or a single incremental count instead of the per-candidate loop), Shiki loading only the languages a response actually contains, block-level memoized markdown so a streamed frame re-renders only the block that changed, and one shared observer for the citation links.

### Acceptance criteria

- A performance profile of one search with AI response and page content enabled, recorded before and after, showing which long tasks were removed.
- Page content tokenization no longer blocks the main thread at the search-to-generation handoff.
- `updateRollingSummary()` encodes each part once instead of re-encoding the joined candidate on every iteration.
- Shiki loads only the languages present in the rendered answer.
- Streaming a long answer re-renders only the markdown block that changed.
- One resize observer for all citation links, and the expand animation uses `transform` and `opacity` only.
- An E2E check that fails if a task longer than 200 ms appears on the main thread while an answer streams.

### Implementation notes

Size: **L**. Each item is independent, so one PR per item.

- `client/modules/textGenerationUtilities.ts` - move the `allocatePageExcerpts()` tokenization into a worker, or budget by character count and tokenize only the trimmed result.
- `client/modules/textGeneration.ts` - keep a running token count in `updateRollingSummary()`.
- `client/modules/shiki.ts` - pass the languages found in the content, or switch to `shiki/bundle/web`.
- `client/components/AiResponse/MarkdownRenderer.tsx` - split the content into blocks and memoize each one.
- `client/components/AiResponse/ExpandableLink.tsx` - one shared `ResizeObserver`, and animate `transform` instead of `width`.
- `e2e/smoke.spec.ts` or a new spec - the long-task assertion, via `PerformanceObserver` on `longtask`.

References:
- [The Browser's Main Thread Is Expensive](https://kciter.so/posts/the-expensive-main-thread/en/)

Contributor guide

Open the contributing guide

Research direction

Start with a performance profile of the search and streaming flow, then read the named entry points in client/modules/textGenerationUtilities.ts, client/modules/textGeneration.ts, client/modules/shiki.ts, MarkdownRenderer.tsx, and ExpandableLink.tsx. Treat each implementation note as an independent change and use e2e/smoke.spec.ts for the long-task check. Done means the listed main-thread, tokenization, highlighting, markdown, resize, animation, and streaming criteria are verified before and after profiling.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
frontend, performance, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.