dotnet / dotnet/fsharp

VS F# editor: race in Tokenizer `SourceTextData` cache can corrupt per-line token data under concurrent access

Open
#20,112 1 comment 0 reactions 0 assignees View on GitHub
Needs-Triage
Dominant language
F#
Stars
4.3k
Forks
876
Avg merge
4d 22h
Merged PRs (30d)
144

Description

Please provide a succinct description of the issue.

`Tokenizer` stores per-line tokenization/classification state in a shared per-document cache. That cache is read and updated from concurrent editor paths (e.g., classification/tagging + symbol lookup). With non-thread-safe storage this can race and produce inconsistent/corrupted cache state.

**Repro steps**

Provide the steps required to reproduce the problem:

1. Open a medium/large F# file in Visual Studio.
2. Trigger concurrent editor activity (rapid edits/scrolling, semantic classification refreshes, and repeated symbol-at-caret requests).
3. Observe intermittent failures or unstable behavior in token/classification flows (historically seen as index/cache inconsistency under contention).

If possible attach a zip file with the repro case. This often makes it easier for others to reproduce.
The zip file should ideally represent the situation just before the call/step that is problematic.

**Expected behavior**

Per-line tokenization cache should remain consistent under concurrent reads/writes and never corrupt internal state.

**Actual behavior**

Shared cache operations can race under concurrent callers and may lead to inconsistent line cache state and downstream failures.

**Known workarounds**

No reliable user workaround; reducing concurrent editor activity can make repro less frequent.

**Related information**

* Operating system: Windows
* .NET Runtime kind (.NET Framework, .NET 8, .NET 10 in repo targets)
* Editing Tools: Visual Studio 2026 Insiders (18.9)

---

## Proposed fix (based on current Tokenizer changes)

In `vsintegration/src/FSharp.Editor/LanguageService/Tokenizer.fs`:

1. Replace non-thread-safe per-line storage with thread-safe storage in `SourceTextData`:
- use `ConcurrentDictionary`
- keep `Item` getter/setter semantics via `TryGetValue`, index assignment, `TryRemove`
2. Keep explicit invalidation API `ClearFrom(n)` that removes cached lines from `n` forward.
3. Preserve existing call sites that invalidate after edits (e.g., `sourceTextDataCache.ClearFrom(endLine + 1)`).
4. Keep per-document cache as `ConcurrentDictionary` so cache entries are safe across concurrent access.

This keeps behavior the same functionally, while making per-line cache mutation safe when multiple editor features touch the same document cache concurrently.

Contributor guide

Open the contributing guide

Research direction

Start in vsintegration/src/FSharp.Editor/LanguageService/Tokenizer.fs, reading SourceTextData and the per-document cache along with the existing ClearFrom call sites. Reproduce concurrent editor activity with a medium or large F# file, then verify that concurrent cache access remains consistent and per-line invalidation still works without downstream tokenization or classification failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
fsharp
Domain
developer-experience, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.