Dealing with ambiguity in the lexical part of the grammar

The procedure defined in §20.4.1.2 (fls_qpx6lgapce57) Token Matching assumes that the contents of a DelimitedTokenTree can be treated as a well-defined sequence of TokenTrees.

The lexical part of the FLS's grammar is ambiguous in many ways, and the discussion of the syntax notation in §1.1.4 (fls_79rl6ylmct07) doesn't say how to resolve these ambiguities.

Consider this MacroInvocation:

m!(ab)

a, b, and ab are all accepted by the Identifier production, and so they're accepted by LexicalElement, NonDelimitedToken, and TokenTree.

That means the DelimitedTokenTree (ab) could be analysed as containing either one identifier ab or two identifiers a and b, leaving the token matching procedure ill-defined.

Of course there are many other examples (1.2 could be 1 . 2, and so on).

The FLS needs to say somewhere how ambiguities are resolved.

(The Reference is in the same position, so it is not a useful guide here.)

The usual thing is to adopt some form of longest-match principle, perhaps essentially saying that at all points the parse that leads to the longest next LexicalElement is preferred.

I don't think that would entirely match how rustc behaves (depending on how other existing issues are resolved), but it would get nearly all the way there.

The first branch of the FloatLiteral production might need a follow restriction to make sure that 1..2 isn't analysed as 1. . 2.

Start with §1.1.4, §20.4.1.2 (fls_qpx6lgapce57) and the Token Matching procedure. Compare the ambiguity examples, including m!(ab), 1.2, and 1..2, with rustc behavior; done means the FLS explicitly defines how lexical ambiguities are resolved and addresses any needed FloatLiteral follow restriction.