Character and string token definitions need updating.

Open
#626 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Stale
Tech stack
rust
Domain
documentation

Research direction

Start by comparing the listed STRING_LITERAL, CHAR_LITERAL, RAW_STRING_LITERAL, BYTE_LITERAL, BYTE_STRING_LITERAL, and RAW_BYTE_STRING_CONTENT definitions with the Lexer definition and the changes in PR 60793. Review the unchecked restrictions and confirm that the English text is synchronized; done means the token descriptions cover the lexer rules and known CRLF behavior without the listed typo.

Written by the indexing model from the issue text.

Description

A-lexer

There are multiple issues here. Some of this has changed in 1.37 via https://github.com/rust-lang/rust/pull/60793.

  • RAW_BYTE_STRING_LITERAL no longer allows bare CR (new 1.37). #1459

  • "Raw string" and "raw byte string" needs to be updated that CRLF is converted to LF (new 1.37). #1459

  • Several tokens need to sync the English text with the "Lexer" definition.

    • STRING_LITERAL indicates several rules (like isolated CR's are not allowed), but the text does not mention any of those restrictions.
    • CHAR_LITERAL says "single Unicode character…except U+0027" which is not complete.
    • RAW_STRING_LITERAL does not allow bare CR's.
    • BYTE_LITERAL escapes are not described.
    • BYTE_STRING_LITERAL restrictions are not described.
    • In general, just make sure they are all in sync!

  • Typo in RAW_BYTE_STRING_CONTENT, points to RAW_STRING_CONTENT when it should be RAW_BYTE_STRING_CONTENT. #818

  • I cannot find anywhere that mentions CRLF in a string is converted to LF. Am I blind? #1459

  • The description for string continuations says "\ immediately before U+000A", but it can also be before CRLF. How should this be handled? I haven't looked at how it is implemented, but are all CRLF's translated everywhere? Should there just be a blanket statement somewhere about this, to avoid having to discuss it in every string literal definition? #1459

I may be missing some things here. Need to very thoroughly review everything to make sure it is correct and up-to-date with the changes from 60793.

Dominant language
Rust
Stars
1.6k
Forks
607
Avg merge
1d 1h
Merged PRs (30d)
12

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from rust-lang/reference

All issues in rust-lang/reference

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.