microsoft / microsoft/vscode-textmate
JSON key `"id"` breaks the tokenizer
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 678
- Forks
- 135
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 2
Description
The internal tokenizer uses the key "id" as a way to keep track of rules within a grammar
However, the parser does not strip "id" keys from the grammar before parsing
if a invalid "id" key is hit during tokenizing, it will lead to unpredictable tokenizing and the grammar partially failing
Luckily it seems no one has placed one in the wrong spot (having one at root is fine)
tho I assume that's because they would have noticed their grammar no longer working when it is present
https://github.com/search?q=path%3A*.tmLanguage.json+%22%5C%22id%5C%22%3A%22&type=code
{
"scopeName": "source.lang",
"patterns": [
{
"begin": "\"",
"end": "\"",
"name": "string",
"patterns": [ { "include": "#backslash" } ]
}
],
"repository": {
"backslash": {
"match": "\\\\.",
"name": "constant.character.escape",
"id": "JSON key \"id\" breaks this rule and its parent"
}
}
}
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named. Start by tracing the grammar parser and internal tokenizer handling of "id" keys, using the JSON grammar example as a reproduction. Add a regression case showing that an "id" inside a rule does not corrupt tokenization or cause the grammar to partially fail.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100