microsoft / microsoft/vscode-textmate
possible unexpected behavior from capture sub-patterns (sub-tokenized capture)
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 678
- Forks
- 135
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 2
Description
I was analyzing the code and noticed that there is no way to anchor (at start) a pattern inside a capture pattern (sub-tokenizing of a capture).
I had not yet the need to do so in any work so far, but had thought that the capture was reprocessed as if it was its own string. So what I notice here is that:
- $ anchor works because the reprocessed string is a substr (0,x) of the original line, ending at the end of the capture.
- \G, \A, and ^ anchors have limited application.
- \G will never match at the start of the sub-tokenized capture. (even if the capture starts at the end of the last BEGIN rule, from what I can tell)
- \A will only match if the sub-tokenized capture is at the start of the first line of the file
- ^ only matches if the sub-tokenized capture is at the start of the current line
- Look behind at the start will work
- look ahead at the end will not work.
- I'm not even going to touch on $Z/$z, as they are outside typical use.
This makes no sense to me. If there was a clear document that stated this was how it was going to work (for all TextMate based products), I'd have to accept it, but since I cannot find one, and when I try to apply logic, I cannot understand why someone would want the anchors to continue their 'document wide' purpose when sub-tokenizing a capture. There is no reason I can imagine where inside a capture I would want to test to see if this point is the first character of the document. I would have done that before capturing the group for sub-tokenizing. It also makes no sense to me to be able to look behind, and not ahead (past end of capture), but I had expected no ability to see past either edges of the capture. I had expected the ^ would match the start of the capture. I had also figured that since \A's description is to match the start of a 'string' that it would also be valid when retokenizing the capture. I have used $ in capture retokenizing, but only so there was something in an END rule, to make it correct.
FYI at https://github.com/Microsoft/vscode-textmate/blob/6b4f4d7bfde2c1baaf8b25906657fd6d5075a6a3/src/grammar.ts#L540-L544, the && captureIndex.start === 0 is redundant. :)
I am not confirming that anything is broken at this time.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/grammar.ts at lines 540-544 and inspect how capture retokenization handles anchors, then compare that behavior with the TextMate grammar documentation. A complete contribution would establish the intended anchor semantics and document them clearly, or identify a specific behavior change supported by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- tooling
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100