microsoft / microsoft/vscode-textmate

possible unexpected behavior from capture sub-patterns (sub-tokenized capture)

Open
#74 1 comment 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature-request
Dominant language
TypeScript
Stars
678
Forks
135
Avg merge
1d 14h
Merged PRs (30d)
2

Description

I was analyzing the code and noticed that there is no way to anchor (at start) a pattern inside a capture pattern (sub-tokenizing of a capture).

I had not yet the need to do so in any work so far, but had thought that the capture was reprocessed as if it was its own string. So what I notice here is that:

  1. $ anchor works because the reprocessed string is a substr (0,x) of the original line, ending at the end of the capture.
  2. \G, \A, and ^ anchors have limited application.
    1. \G will never match at the start of the sub-tokenized capture. (even if the capture starts at the end of the last BEGIN rule, from what I can tell)
    2. \A will only match if the sub-tokenized capture is at the start of the first line of the file
    3. ^ only matches if the sub-tokenized capture is at the start of the current line
  3. Look behind at the start will work
  4. look ahead at the end will not work.
  5. I'm not even going to touch on $Z/$z, as they are outside typical use.

This makes no sense to me. If there was a clear document that stated this was how it was going to work (for all TextMate based products), I'd have to accept it, but since I cannot find one, and when I try to apply logic, I cannot understand why someone would want the anchors to continue their 'document wide' purpose when sub-tokenizing a capture. There is no reason I can imagine where inside a capture I would want to test to see if this point is the first character of the document. I would have done that before capturing the group for sub-tokenizing. It also makes no sense to me to be able to look behind, and not ahead (past end of capture), but I had expected no ability to see past either edges of the capture. I had expected the ^ would match the start of the capture. I had also figured that since \A's description is to match the start of a 'string' that it would also be valid when retokenizing the capture. I have used $ in capture retokenizing, but only so there was something in an END rule, to make it correct.

FYI at https://github.com/Microsoft/vscode-textmate/blob/6b4f4d7bfde2c1baaf8b25906657fd6d5075a6a3/src/grammar.ts#L540-L544, the && captureIndex.start === 0 is redundant. :)

I am not confirming that anything is broken at this time.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/grammar.ts at lines 540-544 and inspect how capture retokenization handles anchors, then compare that behavior with the TextMate grammar documentation. A complete contribution would establish the intended anchor semantics and document them clearly, or identify a specific behavior change supported by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
tooling
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.