microsoft / microsoft/vscode-textmate

Textmate engine bug for `\k<>` backreferences

Open
#193 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
678
Forks
135
Avg merge
1d 14h
Merged PRs (30d)
2

Description

Example of working as expected

Input is on the left, output is on the right.
The "end" pattern is referencing the 2nd group created in "begin" (the EOF)
should_happen

What (intentional) failure looks like (non-issue)

A bad pattern causes this kind of behavior:
(note: yellow is the theme's color for entity.shell, which is the included pattern)
bad_pattern

What is broken

\k<2> should be equivlent to \2 and in other places it does behave equivlently

However, instead of failing normally (e.g. all-yellow) it seems to trigger undefined behavior:
(Note: \2 is not a viable workaround when group numbers are ≥10)
Screen Shot 2022-12-12 at 3 05 46 PM

Here's the code for the problematic pattern.
This is for VS Code 1.72.2, on Mac M1

{
    "begin": "(<<)\\s*+\\s*+((?<!\\w)[a-zA-Z_][a-zA-Z_0-9]*(?!\\w))(?=\\s|;|&|<|\"|')",
    "end": "\\2",
    "beginCaptures": {
        "1": {
            "name": "keyword.operator.heredoc.shell"
        },
        "2": {
            "name": "string.delimiter.shell"
        }
    },
    "endCaptures": {},
    "name": "string.unquoted.heredoc.no-indent.shell",
    "patterns": [
        {
            "match": ".+",
            "name": "entity.shell"
        }
    ]
}

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied begin/end grammar, comparing the \k<2> and \2 backreferences in the TextMate engine. Trace how begin and end backreferences are parsed and handled, including group numbers of 10 or more. Done means invalid backreferences fail normally instead of causing the reported undefined behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.