Add UTF-8 lexer cursor and identifier regression tests
- Dominant language
- Rust
- Stars
- 53
- Forks
- 16
- Avg merge
- 4h 22m
- Merged PRs (30d)
- 46
Description
## Why
The lexer stores byte offsets internally while diagnostic columns count Unicode scalar values. `front/lexer/src/cursor.rs` documents this invariant, but no tests verify it.
## Scope
- Test identifiers containing non-ASCII alphabetic characters.
- Test lookahead and tokenization when multi-byte characters appear before operators and literals.
- Trigger a lexer diagnostic after non-ASCII text and assert the human column rather than the byte offset.
- Cover a newline after UTF-8 input to verify `line_start` is reset correctly.
- Do not introduce Unicode normalization or change identifier policy.
## Likely files
- `front/lexer/src/cursor.rs`
- `front/lexer/src/ident.rs`
- A new lexer integration test
## Completion criteria
- [ ] Multi-byte UTF-8 input never panics or slices at an invalid boundary.
- [ ] Identifier tokens preserve their original spelling.
- [ ] Diagnostic line and column assertions distinguish characters from bytes.
- [ ] `cargo test -p lexer --locked` passes.
Contributor guide
Research direction
Read the UTF-8 offset invariant in front/lexer/src/cursor.rs and identifier handling in front/lexer/src/ident.rs, then inspect the existing lexer test layout. Add integration coverage for non-ASCII identifiers, lookahead, operators and literals, diagnostics after UTF-8 text, and newline line_start handling. Done means the listed assertions pass without invalid-boundary panics; run cargo test -p lexer --locked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- compilers, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100