antlr / antlr/antlr4

New lexer context option proposal: emitOnMatch=true ?

Open
#491 5 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
19k
Forks
3.5k
PR merge metrics
No merged PRs in 30d

Description

As I understand, the lexer choose the first lexer rule that perfectly
matches the longest string it can do with incoming characters.

1st case:
The lexer reads HELLO:
Rule 'HELLO' -> 5 characters matched
Rule [A-Z]+ -> 5 characters matched
Rule 1 WINS, because it is the first rule in the lexer

2nd case:
The lexer reads HELLOS:
Rule 'HELLO' -> 5 characters matched
Rule [A-Z]+ -> 6 characters matched
Rule 2 WINS, because it consumes more characters

I would be useful to have a context option (like greedy) that tell the
lexer to stop trying to find a longer match. In the second case that could
make the rule 1 win.

In my example I would like the lexer to identify special strings that
contains files, and a default string if no special string identified .

```
"C:\share\archive\toto.txt" => QUOTE VOLUME DIRFILE DIRFILE DIRFILE QUOTE
"\\server\archive\toto.txt" => QUOTE SERVER DIRFILE QUOTE
"Any text with characters like : or \" => QUOTE STRING_VALUE QUOTE
```

The rules would be:

```
QUOTE : '"';
VOLUME : [a-zA-Z] ':'
DIRFILE: '\' FILECHARS;
SERVER : '\\' FILECHARS;
STRING_VALUE : ~["]+;
```

The problem is VOLUME 'C:' and SERVER '\server' cannot be matched because of the default STRING_VALUE, that matches a longer string ( C:\toto.txt ) than VOLUME
(C:). So I will allways have the tokens QUOTE STRING_VALUE QUOTE transmitted to the parser. If I add the option on the two rules that are located BEFORE the STRING_VALUE rule:

```
VOLUME : ( options emitOnMatch =true : [a-zA-Z] ':' );
DIRFILE: ( options emitOnMatch =true : '\' FILECHARS);
SERVER : ( options emitOnMatch =true : '\\' FILECHARS);
```

It's an idea for a solution to parse the case above. It would also allow the lexer to process some bytecode languages where is no separators between keywords, where keywords lengths vary, and where there are data values to distinguish from keywords.

Contributor guide

Open the contributing guide

Research direction

No file, test, or implementation entry point is named. Start by reviewing ANTLR's lexer longest-match behavior and the proposed grammar examples, then determine the required semantics and scope for an emitOnMatch option. Done would require an agreed design, implementation, and coverage for the described competing-rule cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
compilers, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.