microsoft / microsoft/vscode-cpptools
Missing semantic tokens after editing in large files
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 6.2k
- Forks
- 1.7k
- Avg merge
- 14h 46m
- Merged PRs (30d)
- 61
Description
If you have a very large file (>5000 lines), an edit may result in temporarily losing semantic tokens up to about 5000 lines before the edits and beyond 5000 lines after the edits. Semantic tokens are restored once a full parse occurs, which occurs when saving the file or switching between documents.
In the short term, we could mitigate the issue by preserving stale tokens instead of clearing them (this would require doing some merging and fixing up positions, at lease for the tokens after the edits). However, to keep all of the semantic tokens in the file always up to date, the following issue may be a prerequisite: https://github.com/microsoft/vscode-cpptools/issues/7759
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin by investigating the semantic-token handling for edits in files over 5,000 lines and reproduce the loss of tokens before and after an edit. Compare the short-term stale-token mitigation with the prerequisite issue #7759, and consider the work complete when tokens remain correct without requiring a save or document switch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, typescript
- Domain
- devtools, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100