dtolnay / dtolnay/proc-macro2

Fallback `Ident` is not normalized

Open
#500 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
934
Forks
135
PR merge metrics
No merged PRs in 30d

Description

Starting from Rust 1.42.0 (rust-lang/rust#66670) Unicode identifiers are normalized in NFC form, that means the following proc-macro will produce `true`:

```rust
extern crate proc_macro;
use proc_macro::{Ident, Span, TokenStream};

#[proc_macro]
pub fn check(_: TokenStream) -> TokenStream {
let decomposed = Ident::new("e\u{301}", Span::call_site());
let composed = Ident::new("\u{e9}", Span::call_site());
(composed.to_string() == decomposed.to_string()).to_string().parse().unwrap()
}
```

However, the `proc_macro2` fallback does not perform NFC normalization, meaning the equivalent code in `proc_macro2` will produce `false`:

```rust
use proc_macro2::{Ident, Span, TokenStream};

pub fn check(_: TokenStream) -> TokenStream {
let decomposed = Ident::new("e\u{301}", Span::call_site());
let composed = Ident::new("\u{e9}", Span::call_site());
(composed.to_string() == decomposed.to_string()).to_string().parse().unwrap()
}

fn main() {
dbg!(check(TokenStream::default()));
// Ident { sym: false, span: bytes(1..6) }
}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.