microsoft / microsoft/vscode-textmate
Capturing and applying a pattern causes performance loss
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 678
- Forks
- 135
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 2
Description
Create a syntax highlighter with the following code:

{
"scopeName": "source.redcmd.syntax.lag",
"patterns": [
{
"match": "a",
"name": "keyword.control",
"captures": {
"0": { "patterns": [] }
}
}
]
}
Create a test file, apply that syntax highlighter and type a few thousand a's
notice the increasing lag and vscode eventually giving up trying to match and just highlighting the rest of the file with the last token

(you may need to increase editor.maxTokenizationLineLength to 10,000 or so to see results)
Now remove "patterns": []
notice all lag disappearing and everything tokenizes correctly

Also notice that "patterns": [] can be empty (which would be a different bug in itself) and still causes lag
The capture group itself does need to capture something (does not happen with a empty group)
and the size of the capture text inside the group does not matter
Without "patterns": []; my machine is able to highlight almost 680,000 a's
While with patterns; it can barely highlight 6,500
Thats almost exactly 100x times faster
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with the shown grammar, an applied syntax highlighter, and a line containing thousands of a characters, comparing the cases with and without "patterns": []. Trace the tokenization path handling captured text and nested patterns. Done means the grammar tokenizes the full line correctly without the reported performance collapse.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- performance, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100