Tokenizing massive F# file leads to LOH allocations and UI freezes in VS
- Dominant language
- F#
- Stars
- 4.3k
- Forks
- 876
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 144
Description
1. Open the F# codebase
2. Open some F# compiler files (this is just to get memory usage up to where it normally is when working in the compiler)
3. Open `ComparersRegression.fs` (nearly 60k LoC)
4. Scroll around, note that memory pressure is high in VS and the UI freezes often
From a 50ish second sample:
These LOH allocs are all coming from tokenizing the source, the `SourceTextData.extendTo` function: https://github.com/dotnet/fsharp/blob/be621d8725e3ea88cb832f4d457bc0bf377172ed/vsintegration/src/FSharp.Editor/LanguageService/Tokenizer.fs#L419
And tokenizing source itself is the lion's share of all allocs. Note the LOH part of it:
Contributor guide
Assessment
This issue has not been assessed yet.