rust-lang / rust-lang/rust

Lexer accidentally(?) does not use is_ascii_whitespace for literal whitespace in string continuations

Open
#136,600 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-parser C-bug T-compiler T-lang
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

https://github.com/rust-lang/rust/pull/108403 proposed to fix this, but it was claimed that the current behavior was documented in the reference in this comment. Incorrectly, as far as I can see, as that page only describes whitespace escapes as being \r, \t, and \n and the fix was about literal whitespace in string continuations. Now https://doc.rust-lang.org/reference/expressions/literal-expr.html#string-continuation-escapes does describe this behavior, but this was added later in Jan 2024. Indeed, this PR shows the reference documented skipping all whitespace, until Jun 13, 2022.

Current behavior has this ui test. It seems like this behavior was once implemented like it is now, then got claimed to be canon then got documented as canon. Anyway, I'm not sure why not all unicode whitespace is skipped, but just almost all ascii whitespace, but it seems important to pick an existing whitespace set, instead of using an old bad manual implementation of is_ascii_whitespace...

Perhaps we can see a crater run at least...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing UI test at tests/ui/str/str-escape.rs and compare its literal string-continuation cases with the documented behavior in the Rust Reference. Trace the lexer handling of literal whitespace and its use of is_ascii_whitespace, then check the linked historical pull requests for the intended behavior. Done means the whitespace set, implementation, test expectations, and reference documentation agree.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.