Make tokenization registries and token palettes scopeable for multi-theme fixtures
- Dominant language
- TypeScript
- Stars
- 193k
- Forks
- 42.4k
- PR merge metrics
- PR metrics pending
Description
## Engineering task
Make editor tokenization registries and token palettes scopeable/injectable so multiple differently themed editor fixtures can coexist in **one JavaScript context** while using VS Code's real TextMate implementation.
This is a future infrastructure task, separate from the multi-diff styling and fixture work that motivated it. Iframe isolation is not a solution for this use case.
## Motivation
Component fixtures need to render Dark, Light, and high-contrast editors together. TextMate's encoded tokens contain palette indices, but editor models and several renderers currently use the process-wide `TokenizationRegistry` and its color map. Switching that global theme per fixture cannot reliably preserve the relationship between each model's tokens and its palette.
Reusing `TMGrammarFactory`, `TextMateTokenizationSupport`, the shipped grammars, and Oniguruma should not require approximating syntax scopes or changing fixture model language IDs. WASM/grammar initialization can be asynchronous; tokenization itself can remain synchronous after initialization.
## Proposed direction
- Introduce explicit, injectable ownership for tokenization registries and token palettes. Preserve the existing default behavior for production consumers and standalone APIs.
- Allow fixture child instantiation scopes to own their registry, TextMate grammar factory, palette, and token CSS while sharing the WASM runtime and loaded grammar assets where appropriate.
- Thread the scope through text-model tokenization and editor rendering. A palette-only service is insufficient if model tokenization still uses another registry.
- Pass dependencies through the existing editor/view context where components are not directly constructed through DI.
- Make the minimap color tracker scope-owned rather than a process-wide singleton.
- Ensure registrations, listeners, factories, and caches have explicit disposal ownership.
Relevant starting points:
- `src/vs/editor/common/languages.ts`: existing `ITokenizationRegistry` contract and global instance.
- `src/vs/editor/common/tokenizationRegistry.ts`: registry and color-map storage.
- `src/vs/editor/common/model/tokens/tokenizerSyntaxTokenBackend.ts`: global tokenizer lookup/change subscription.
- `src/vs/editor/common/viewModel/minimapTokensColorTracker.ts`: global palette cache.
- `src/vs/editor/common/viewModel/viewContext.ts`: dependency propagation to view parts.
- `src/vs/workbench/services/textMate/browser/textMateTokenizationFeatureImpl.ts`: TextMate registration and palette publication.
## Audited palette consumers
In the motivating checkout, there are 18 files with 20 direct palette-read sites: 15 runtime files and 3 development/fixture files.
Runtime uses include:
- Minimap colors and overview-ruler background/change handling.
- Clipboard HTML, IME composition text, and token inspection.
- Token CSS for extensions, MCP/plugins, release notes, walkthroughs, notebooks, and mobile diffs.
- Notebook drag previews and extension-API syntax-highlighting results.
Also account for indirect minimap consumers and change-event-only consumers such as hover rerendering. The two production palette writers are `StandaloneThemeService` and `TextMateTokenizationFeature`; existing fixture writers need migration too.
## Acceptance criteria
- Dark, Light, and both high-contrast fixtures can render concurrently in one JS context with real TextMate tokenization and no cross-fixture palette interference.
- Original language IDs, grammar injections, token-type metadata, and language-feature behavior are preserved.
- Text, font styles, minimap/canvas rendering, clipboard/IME colors, and other palette-dependent paths use the matching scope.
- Changing or disposing one fixture's theme/registry does not retokenize, recolor, or break another fixture.
- Existing production and standalone behavior is preserved, including lazy tokenizer registration and palette-change notifications.
- Targeted tests cover simultaneous themes, differing render/disposal orders, updates, and leak-free teardown.
Semantic highlighting from language-service providers is a separate concern; it should not be conflated with TextMate syntax highlighting in this task.
Contributor guide
Assessment
This issue has not been assessed yet.