microsoft / microsoft/vscode-textmate

TextMate lacks means to match a pattern exactly one time

Open
#114 5 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature-request
Dominant language
TypeScript
Stars
678
Forks
135
Avg merge
1d 14h
Merged PRs (30d)
2

Description

TextMate seems to lack a means to indicate that a pattern is to be matched only one time, similar to the REGEX ? quantifier. Without this, it makes some languages difficult to correctly scope.

Take for instance PowerShell, which doesn't really have any reserved keywords. PowerShell does support if, elseif and else keywords for flow control, and like many languages, else is only supported once after an if statement. However, because keywords are not reserved, elseif and else can be reused as command names, and the only factor that controls that, is context. While in the context of an if statement, elseif and else serve as keywords, with else or anything that is not elseif terminating the if context. Additionally, () condition group and {} statement blocks are required for if and elseif and {} is required for else (but that is not that important since the if context is already terminated effectively).

# general syntax (not including comments and line breaks)
if (condition) {statement} elseif (condition-elseif) {statement-elseif} else {statement-else}

# demonstration
# `else another-command` is actually another command (native executable or user defined function)
if(1){do-something}else{do-something else}else another-command

To further complicate things, the state at which this statement is reached is unknown, as there are multiple ways to arrive at it. This prevents having a forceful exit strategy where by the grammar can force a specific rule to progress until it is clear to back out of the current stack.

Example

# hashtable using `if` in assignment
@{
    key = if (cond) {statement}
    key2 = if (cond) {statement} else {statement else}
    else = 'const' # else here is valid hashtable literal key name, because `if` context ended above.
    key3 = if (condition) {statement}
    else {another statement} # `if ` context was not closed in previous line so it continued to this line
}

(Note the grammar that GitHub uses only differentiates the else hash key above based on the presence of an =, not on the context of the if statement.)

I cannot find a means to formulate a TextMate grammer than can properly describe this situation. I think this is due to the lack of a property on each begin or match rule such as applyPatternOnce, which would limit the matching of the pattern to only once in the current stack scope.

  • else should only be allowed once per if context.
  • () condition should be required but only once per if or elseif subcontext.
  • {} statement block should be required, but only once per if, elseif or else subcontext, but only after the () condition for if and elseif.

Grammar constructed so far: (only partial file) (if testing, only use empty conditions and empty statement blocks and no comments, as I am not including those subsections as they are not relevant to this issue.)

{
	"patterns": [
		{
			"comment": "else,elseif: only after if,elseif",
			"begin": "(?i)(?=if[\\s{(,;&|)}])",
			"end": "(?!\\G)",
			"patterns": [
				{
					"include": "#ifStatement"
				}
			]
		}
	],
	"repository": {
		"ifStatement": {
			"comment": "else,elseif: only after if,elseif",
			"begin": "\\G(?i:(if)|(elseif))(?=[\\s{(,;&|)}])",
			"beginCaptures": {
				"1": {
					"name": "keyword.control.if.powershell"
				},
				"2": {
					"name": "keyword.control.if-elseif.powershell"
				}
			},
			"end": "(?=.|$)",
			"applyEndPatternLast": true,
			"patterns": [
				{
					"include": "#advanceToToken"
				},
				{
					"begin": "(?<![)}])(?=\\()",
					"end": "(?=.|$)",
					"applyEndPatternLast": true,
					"patterns": [
						{
							"begin": "\\G\\(",
							"beginCaptures": {
								"0": {
									"name": "punctuation.section.group.begin.powershell"
								}
							},
							"end": "\\)",
							"endCaptures": {
								"0": {
									"name": "punctuation.section.group.end.powershell"
								}
							},
							"name": "meta.if-condition.powershell",
							"patterns": [
								{
									"comment": "`;` not permitted here",
									"match": ";",
									"name": "invalid.source.powershell"
								},
								{
									"include": "#command_mode"
								}
							]
						},
						{
							"begin": "(?<=\\))(?=[\\s#]|<#|`\\s|{)",
							"end": "(?=.|$)",
							"applyEndPatternLast": true,
							"patterns": [
								{
									"include": "#advanceToToken"
								},
								{
									"begin": "(?<!})(?={)",
									"end": "(?=.|$)",
									"applyEndPatternLast": true,
									"patterns": [
										{
											"begin": "\\G\\{",
											"beginCaptures": {
												"0": {
													"name": "punctuation.section.braces.begin.powershell"
												}
											},
											"end": "}",
											"endCaptures": {
												"0": {
													"name": "punctuation.section.braces.end.powershell"
												}
											},
											"name": "meta.statements.if-condition.powershell",
											"patterns": [
												{
													"include": "$self"
												}
											]
										},
										{
											"begin": "(?<=})(?=[\\s#]|<#|`\\s)",
											"end": "(?=.|$)",
											"applyEndPatternLast": true,
											"patterns": [
												{
													"include": "#advanceToToken"
												}
											]
										}
									]
								}
							]
						}
					]
				},
				{
					"begin": "(?i:else)(?=[\\s{(,;&|)}])",
					"beginCaptures": {
						"0": {
							"name": "keyword.control.if-else.powershell"
						}
					},
					"end": "(?=.|$)",
					"applyEndPatternLast": true,
					"patterns": [
						{
							"include": "#advanceToToken"
						},
						{
							"begin": "(?<!}){",
							"beginCaptures": {
								"0": {
									"name": "punctuation.section.braces.begin.powershell"
								}
							},
							"end": "}",
							"endCaptures": {
								"0": {
									"name": "punctuation.section.braces.end.powershell"
								}
							},
							"name": "meta.statements.if-else-condition.powershell",
							"patterns": [
								{
									"include": "$self"
								}
							]
						}
					]
				},
				{
					"begin": "(?i)(?=elseif[\\s{(,;&|)}])",
					"end": "(?!\\G)",
					"patterns": [
						{
							"include": "#ifStatement"
						}
					]
				}
			]
		},
		"advanceToToken": {
			"comment": "consume spaces and comments and line ends until the next token appears",
			"begin": "\\G(?=[\\s#]|<#|`\\s)",
			"end": "(?!\\s)(?!$)",
			"applyEndPatternLast": true,
			"patterns": [
				{
					"comment": "useless escape, and doesn't count as a token",
					"match": "`\\s",
					"name": "invalid.character.escape.powershell"
				},
				{
					"include": "#commentLine"
				},
				{
					"include": "#commentBlock"
				}
			]
		}
	}
}

For reference to this grammar (the complete grammar): https://github.com/msftrncs/PowerShell.tmLanguage/blob/argumentmode_2ndtry/powershell.tmLanguage.json

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the requested applyPatternOnce behavior and the complete PowerShell.tmLanguage.json linked in the issue, using its if/elseif/else examples as the reproduction. Identify how a rule could be limited to one match in the current stack scope; done means the grammar can distinguish a single else, condition group, and statement block as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.