dtolnay / dtolnay/proc-macro2

Carriage return in string continuation doesn't match rustc's behavior

Open
#540 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
934
Forks
135
PR merge metrics
No merged PRs in 30d

Description

proc-macro2 seems to reject carriage returns at the end of a string continuation when not followed by a newline. However, rustc seems to accept this (and eats the carriage return).

```rust
use std::str::FromStr;
fn main() {
// This generates a LexError.
println!("{:?}", proc_macro2::TokenStream::from_str("\"\\\n\r\""));
}
```

In rustc, this is treated as an empty string. All TAB, LF, CR, and SP characters are removed after the opening backslash-newline.

This was added in https://github.com/dtolnay/proc-macro2/commit/638a1336b203afe8c669429e21fd3d4e1797e090 (and changed in https://github.com/dtolnay/proc-macro2/commit/72f39fa6004dd22f69d2e254999bd34049f0e69a), but it is not clear why that check was added.

The code for handling continuations in rustc I believe is around here: https://github.com/rust-lang/literal-escaper/blob/ccd015ab0b388fb5e1c0c1085307aedd8210c7c8/src/lib.rs#L474-L500

Contributor guide

No contributing guide indexed for this repository

Research direction

Run the provided Rust reproduction through proc_macro2::TokenStream::from_str to confirm the LexError. Compare its continuation handling with the referenced rustc implementation in literal-escaper/src/lib.rs around lines 474-500. Done means a carriage return at the end of a string continuation is accepted and consumed consistently with rustc.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.