antlr / antlr/antlr4

Lexical mode problem with sentinel character?

Open
#982 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
19k
Forks
3.5k
PR merge metrics
No merged PRs in 30d

Description

The Islands in the Stream section in the ANTLR book discusses lexical mode switching and uses XML as an example. The lexer has the following:

```
lexer grammar ModeTagsLexer;

// Default mode rules (the SEA)
OPEN : '<' -> mode(ISLAND) ; // switch to ISLAND mode
TEXT : ~'<'+ ; // clump all text together

mode ISLAND;
CLOSE : '>' -> mode(DEFAULT_MODE) ; // back to SEA mode
SLASH : '/' ;
ID : [a-zA-Z]+ ; // match/send ID in tag to parser
```

The TEXT lexer rule includes all text _except_ the < character. But what if your text inside the XML needs to have that character? Is this possible?

Contributor guide

Open the contributing guide

Research direction

Start with the ModeTagsLexer grammar and the Islands in the Stream section cited in the issue. Determine whether text containing '<' can be represented while preserving the shown lexical mode switching, and document a clear answer or example as the completed outcome.

Written by the indexing model from the issue text.

Assessment

Domain
compilers
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.