VS F# editor: race in Tokenizer `SourceTextData` cache can corrupt per-line token data under concurrent access
- Dominant language
- F#
- Stars
- 4.3k
- Forks
- 876
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 144
Description
Please provide a succinct description of the issue.
`Tokenizer` stores per-line tokenization/classification state in a shared per-document cache. That cache is read and updated from concurrent editor paths (e.g., classification/tagging + symbol lookup). With non-thread-safe storage this can race and produce inconsistent/corrupted cache state.
**Repro steps**
Provide the steps required to reproduce the problem:
1. Open a medium/large F# file in Visual Studio.
2. Trigger concurrent editor activity (rapid edits/scrolling, semantic classification refreshes, and repeated symbol-at-caret requests).
3. Observe intermittent failures or unstable behavior in token/classification flows (historically seen as index/cache inconsistency under contention).
If possible attach a zip file with the repro case. This often makes it easier for others to reproduce.
The zip file should ideally represent the situation just before the call/step that is problematic.
**Expected behavior**
Per-line tokenization cache should remain consistent under concurrent reads/writes and never corrupt internal state.
**Actual behavior**
Shared cache operations can race under concurrent callers and may lead to inconsistent line cache state and downstream failures.
**Known workarounds**
No reliable user workaround; reducing concurrent editor activity can make repro less frequent.
**Related information**
* Operating system: Windows
* .NET Runtime kind (.NET Framework, .NET 8, .NET 10 in repo targets)
* Editing Tools: Visual Studio 2026 Insiders (18.9)
---
## Proposed fix (based on current Tokenizer changes)
In `vsintegration/src/FSharp.Editor/LanguageService/Tokenizer.fs`:
1. Replace non-thread-safe per-line storage with thread-safe storage in `SourceTextData`:
- use `ConcurrentDictionary`
- keep `Item` getter/setter semantics via `TryGetValue`, index assignment, `TryRemove`
2. Keep explicit invalidation API `ClearFrom(n)` that removes cached lines from `n` forward.
3. Preserve existing call sites that invalidate after edits (e.g., `sourceTextDataCache.ClearFrom(endLine + 1)`).
4. Keep per-document cache as `ConcurrentDictionary` so cache entries are safe across concurrent access.
This keeps behavior the same functionally, while making per-line cache mutation safe when multiple editor features touch the same document cache concurrently.
Contributor guide
Research direction
Start in vsintegration/src/FSharp.Editor/LanguageService/Tokenizer.fs, reading SourceTextData and the per-document cache along with the existing ClearFrom call sites. Reproduce concurrent editor activity with a medium or large F# file, then verify that concurrent cache access remains consistent and per-line invalidation still works without downstream tokenization or classification failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fsharp
- Domain
- developer-experience, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 64/100