microsoft / microsoft/vscode-textmate

Capturing and applying a pattern causes performance loss

Open
#167 0 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
678
Forks
135
Avg merge
1d 14h
Merged PRs (30d)
2

Description

Create a syntax highlighter with the following code:
image

{
	"scopeName": "source.redcmd.syntax.lag",
	"patterns": [
		{
			"match": "a",
			"name": "keyword.control",
			"captures": {
				"0": { "patterns": [] }
			}
		}
	]
}

Create a test file, apply that syntax highlighter and type a few thousand a's
notice the increasing lag and vscode eventually giving up trying to match and just highlighting the rest of the file with the last token
image
(you may need to increase editor.maxTokenizationLineLength to 10,000 or so to see results)

Now remove "patterns": []
notice all lag disappearing and everything tokenizes correctly
image

Also notice that "patterns": [] can be empty (which would be a different bug in itself) and still causes lag
The capture group itself does need to capture something (does not happen with a empty group)
and the size of the capture text inside the group does not matter

Without "patterns": []; my machine is able to highlight almost 680,000 a's
While with patterns; it can barely highlight 6,500
Thats almost exactly 100x times faster

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the shown grammar, an applied syntax highlighter, and a line containing thousands of a characters, comparing the cases with and without "patterns": []. Trace the tokenization path handling captured text and nested patterns. Done means the grammar tokenizes the full line correctly without the reported performance collapse.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
performance, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.