microsoft / microsoft/vscode

Make tokenization registries and token palettes scopeable for multi-theme fixtures

Open
#335,530 0 comments 0 reactions 1 assignee Claimed by @hediet View on GitHub
Dominant language
TypeScript
Stars
193k
Forks
42.4k
PR merge metrics
PR metrics pending

Description

## Engineering task

Make editor tokenization registries and token palettes scopeable/injectable so multiple differently themed editor fixtures can coexist in **one JavaScript context** while using VS Code's real TextMate implementation.

This is a future infrastructure task, separate from the multi-diff styling and fixture work that motivated it. Iframe isolation is not a solution for this use case.

## Motivation

Component fixtures need to render Dark, Light, and high-contrast editors together. TextMate's encoded tokens contain palette indices, but editor models and several renderers currently use the process-wide `TokenizationRegistry` and its color map. Switching that global theme per fixture cannot reliably preserve the relationship between each model's tokens and its palette.

Reusing `TMGrammarFactory`, `TextMateTokenizationSupport`, the shipped grammars, and Oniguruma should not require approximating syntax scopes or changing fixture model language IDs. WASM/grammar initialization can be asynchronous; tokenization itself can remain synchronous after initialization.

## Proposed direction

- Introduce explicit, injectable ownership for tokenization registries and token palettes. Preserve the existing default behavior for production consumers and standalone APIs.
- Allow fixture child instantiation scopes to own their registry, TextMate grammar factory, palette, and token CSS while sharing the WASM runtime and loaded grammar assets where appropriate.
- Thread the scope through text-model tokenization and editor rendering. A palette-only service is insufficient if model tokenization still uses another registry.
- Pass dependencies through the existing editor/view context where components are not directly constructed through DI.
- Make the minimap color tracker scope-owned rather than a process-wide singleton.
- Ensure registrations, listeners, factories, and caches have explicit disposal ownership.

Relevant starting points:
- `src/vs/editor/common/languages.ts`: existing `ITokenizationRegistry` contract and global instance.
- `src/vs/editor/common/tokenizationRegistry.ts`: registry and color-map storage.
- `src/vs/editor/common/model/tokens/tokenizerSyntaxTokenBackend.ts`: global tokenizer lookup/change subscription.
- `src/vs/editor/common/viewModel/minimapTokensColorTracker.ts`: global palette cache.
- `src/vs/editor/common/viewModel/viewContext.ts`: dependency propagation to view parts.
- `src/vs/workbench/services/textMate/browser/textMateTokenizationFeatureImpl.ts`: TextMate registration and palette publication.

## Audited palette consumers

In the motivating checkout, there are 18 files with 20 direct palette-read sites: 15 runtime files and 3 development/fixture files.

Runtime uses include:
- Minimap colors and overview-ruler background/change handling.
- Clipboard HTML, IME composition text, and token inspection.
- Token CSS for extensions, MCP/plugins, release notes, walkthroughs, notebooks, and mobile diffs.
- Notebook drag previews and extension-API syntax-highlighting results.

Also account for indirect minimap consumers and change-event-only consumers such as hover rerendering. The two production palette writers are `StandaloneThemeService` and `TextMateTokenizationFeature`; existing fixture writers need migration too.

## Acceptance criteria

- Dark, Light, and both high-contrast fixtures can render concurrently in one JS context with real TextMate tokenization and no cross-fixture palette interference.
- Original language IDs, grammar injections, token-type metadata, and language-feature behavior are preserved.
- Text, font styles, minimap/canvas rendering, clipboard/IME colors, and other palette-dependent paths use the matching scope.
- Changing or disposing one fixture's theme/registry does not retokenize, recolor, or break another fixture.
- Existing production and standalone behavior is preserved, including lazy tokenizer registration and palette-change notifications.
- Targeted tests cover simultaneous themes, differing render/disposal orders, updates, and leak-free teardown.

Semantic highlighting from language-service providers is a separate concern; it should not be conflated with TextMate syntax highlighting in this task.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.