microsoft / microsoft/vscode-textmate

JSON key `"id"` breaks the tokenizer

Open
#229 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
678
Forks
135
Avg merge
1d 14h
Merged PRs (30d)
2

Description

The internal tokenizer uses the key "id" as a way to keep track of rules within a grammar
However, the parser does not strip "id" keys from the grammar before parsing
if a invalid "id" key is hit during tokenizing, it will lead to unpredictable tokenizing and the grammar partially failing

Luckily it seems no one has placed one in the wrong spot (having one at root is fine)
tho I assume that's because they would have noticed their grammar no longer working when it is present
https://github.com/search?q=path%3A*.tmLanguage.json+%22%5C%22id%5C%22%3A%22&type=code

{
	"scopeName": "source.lang",
	"patterns": [
		{
			"begin": "\"",
			"end": "\"",
			"name": "string",
			"patterns": [ { "include": "#backslash" } ]
		}
	],
	"repository": {
		"backslash": {
			"match": "\\\\.",
			"name": "constant.character.escape",
			"id": "JSON key \"id\" breaks this rule and its parent"
		}
	}
}

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named. Start by tracing the grammar parser and internal tokenizer handling of "id" keys, using the JSON grammar example as a reproduction. Add a regression case showing that an "id" inside a rule does not corrupt tokenization or cause the grammar to partially fail.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.