rust-lang / rust-lang/rust

Decide what to do about the places where we strip shebang but not frontmatter

Open
#161,461 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-maybe-future-edition A-proc-macros C-bug F-frontmatter T-compiler T-libs
Dominant language
Rust
Stars
119k
Forks
16.2k
PR merge metrics
PR metrics pending

Description

This issue keeps track of the following FIXME so it doesn't get forgotten (esp. since this could be relevant for the next edition):

https://github.com/rust-lang/rust/blob/526c36b4c477994b7905ec6a084d6150e49076f5/compiler/rustc_parse/src/lib.rs#L250-L253

Us not stripping frontmatter (but shebang) in source_str_to_stream for backward compatibility reasons (https://github.com/rust-lang/rust/issues/145520) means (among other things) that we don't strip frontmatter in proc_macro_server::Rustc::ts_from_str aka proc_macro::TokenStream::from_str across the bridge.

As a reminder, RFC 3503 | frontmatter set a goal to strip frontmatter everywhere where shebang is stripped:

This applies anywhere shebang stripping is performed. For example, if include! strips shebangs, then it will also frontmatter.


Two potential solutions:

  1. Start stripping frontmatter in TokenStream::from_str in the next edition
  2. Stop stripping shebang in TokenStream::from_str (†)

(†) Reasoning: What TokenStream::from_str receives represents arbitrary Rust source code (if lexically valid) like (parts of) expressions, patterns, types, items, bounds, etc. Therefore, I would say it's not really understood to be taking an entire source file where it would make sense to perform all transformations (BOM removal, CRLF "normalization", shebang+frontmatter stripping). As a another datapoint, we've recently stopped stripping shebang(+frontmatter) in expression-context include!(…)s (contrary to item context), https://github.com/rust-lang/rust/pull/146377, for a similar reason.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the FIXME in compiler/rustc_parse/src/lib.rs and inspect source_str_to_stream alongside proc_macro_server::Rustc::ts_from_str and TokenStream::from_str. Read RFC 3503, issue 145520, and PR 146377 to compare the two proposed behaviors. Done means a decided and consistently implemented treatment of shebang and frontmatter in the affected paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.