Lexer accidentally(?) does not use is_ascii_whitespace for literal whitespace in string continuations
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 119k
- Forks
- 16.1k
- PR merge metrics
- PR metrics pending
Description
https://github.com/rust-lang/rust/pull/108403 proposed to fix this, but it was claimed that the current behavior was documented in the reference in this comment. Incorrectly, as far as I can see, as that page only describes whitespace escapes as being \r, \t, and \n and the fix was about literal whitespace in string continuations. Now https://doc.rust-lang.org/reference/expressions/literal-expr.html#string-continuation-escapes does describe this behavior, but this was added later in Jan 2024. Indeed, this PR shows the reference documented skipping all whitespace, until Jun 13, 2022.
Current behavior has this ui test. It seems like this behavior was once implemented like it is now, then got claimed to be canon then got documented as canon. Anyway, I'm not sure why not all unicode whitespace is skipped, but just almost all ascii whitespace, but it seems important to pick an existing whitespace set, instead of using an old bad manual implementation of is_ascii_whitespace...
Perhaps we can see a crater run at least...
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing UI test at tests/ui/str/str-escape.rs and compare its literal string-continuation cases with the documented behavior in the Rust Reference. Trace the lexer handling of literal whitespace and its use of is_ascii_whitespace, then check the linked historical pull requests for the intended behavior. Done means the whitespace set, implementation, test expectations, and reference documentation agree.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100