Make CR-only source line endings consistent across lexer tracking and literals
- Dominant language
- Rust
- Stars
- 53
- Forks
- 16
- Avg merge
- 4h 22m
- Merged PRs (30d)
- 46
Description
The lexer recognizes carriage return (`\\r`) as whitespace, but only line feed (`\\n`) advances the logical source line.
In `skip_whitespace`, `\\r` is consumed together with ordinary horizontal whitespace, while `\\n` increments `line` and updates `line_start`.
The literal scanner also treats only `\\n` as a raw source newline terminating a string. A lone `\\r` can therefore be consumed as content even though the same character is treated as source whitespace outside the literal.
This creates inconsistent behavior for CR-only source files:
- token line numbers do not advance at `\\r`,
- columns continue from the previous physical line,
- raw CR inside a string does not take the same newline path as LF,
- diagnostics can disagree with the physical source layout.
CRLF must remain a single newline rather than becoming two logical lines.
Code evidence:
- `front/lexer/src/trivia.rs` — `skip_whitespace`
- `front/lexer/src/literals.rs` — string literal newline handling
- lexer diagnostic line/column tracking
Acceptance:
- [ ] Define source newline handling for LF, CRLF, and lone CR.
- [ ] Count CRLF as exactly one logical newline.
- [ ] Handle lone CR consistently in token tracking, comments, and literals.
- [ ] Add lexer tests comparing equivalent LF, CRLF, and CR-only sources.
- [ ] Verify diagnostic line and column locations under all three encodings.
- [ ] Avoid changing escape-sequence `\\r` semantics inside string/char literals.
Contributor guide
Research direction
Start in front/lexer/src/trivia.rs and front/lexer/src/literals.rs, then trace the lexer diagnostic line/column tracking. Compare LF, CRLF, and lone-CR inputs through whitespace, comments, and literals while preserving escaped \r behavior. Done means each physical newline form produces consistent tokens and diagnostics, with CRLF counted once.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100