Input that must be rejected by the lexer

There are a number of forms which rustc rejects at lexing time but the FLS says are accepted as multiple lexical elements.

Forms similar to numeric literals

In addition to those covered by #602, forms that "look like" the following are rejected:

• a radix prefix with no following digits
• a floating-point literal with no digits in the exponent
• a floating-point literal with a radix prefix.

Some examples:

0x1.2;   // this is not an integer literal followed by `.` and `2`
0b101e;  // this is not a suffixed literal, or `0b101` followed by `e`
0b;      // this is not an integer literal, or `0` followed by `b`
0b_;     // this is not an integer literal, or `0` followed by `b_`
2e;      // this is not a floating-point literal, or `2` followed by `e`
2.0e;    // this is not a floating-point literal, or `2.0` followed by `e`
2em;     // this is not a suffixed literal, or `2` followed by `em`
2.0em;   // this is not a suffixed literal, or `2.0` followed by `em`

The Reference models this as part of lex.token.literal.reserved.syntax, by treating these as reserved forms. That means that, as in #601 and #602, it tries to use grammar productions that don't overlap with the productions for accepted tokens.

I think the forms given in the Reference are close to correct.

There is an ambiguity between FLOAT_LITERAL and RESERVED_NUMBER: a form like 123e_1 matches both.

It gets a form like 1_· wrong (· is a character in XID_Continue but not in XID_Start). rustc rejects that but it matches the Reference's INTEGER_LITERAL.

Prefixed quoted literals

In contrast to #601 and #602, as far as I know there's no RFC describing this.

The general idea is that anything that looks like the prefix of a single- or double-quoted literal must introduce a well-formed literal of the kind indicated by the prefix; otherwise lexing fails.

For example:

• b"€" is rejected, not accepted as Identifer b followed by SimpleStringLiteral "€".

• c"\u{0000}" is rejected, not accepted as Identifer c followed by SimpleStringLiteral "\u{0000}".

• Forms similar to raw quoted literals which exceed the limit of 255 hashes (see #600) are rejected, not accepted as a combination of an Identifier, Punctuation tokens, and a simple literal.

The Reference doesn't try to model these. There's an open issue at rust-lang/reference#626.

I don't think trying to handle these with Reference-style reserved forms which don't overlap with the forms for accepted tokens would come out cleanly.

If #589 is resolved using some form of longest-match principle and #601 is resolved by using reserved forms, I think (as long as the FLS doesn't need to model the 2015 edition) those reserved forms could be made to take care of this case by dropping any special mention of the prefixes b, c, r, br, and cr.

Suffixed quoted literals

Once #599 is addressed, the FLS will need to say that forms like "x"_ where the apparent suffix is a single underscore are rejected, not accepted as SimpleStringLiteral "x" followed by Punctuation _.

Lifetimes and labels

Forms like 'ab'c or 'r#a'b are rejected, not accepted as four tokens (or as two tokens if #582 is resolved by introducing something like the Reference's LIFETIME_TOKEN).

The Reference deals with this with a follow restriction (see lex.token.life.syntax).

Start with the FLS sections lex.token.literal.reserved.syntax and lex.token.life.syntax, then review the related issues #599, #601, #602, #600, #582, and Reference issue #626. Compare the listed lexer-rejected forms with rustc behavior and the existing Reference model. Done means the FLS specifies these rejection cases without incorrectly treating them as multiple accepted tokens.