wavefnd / wavefnd/Wave

Add UTF-8 lexer cursor and identifier regression tests

Open
#392 0 comments 0 reactions 0 assignees View on GitHub
cross-platform good first issue needs testing
Dominant language
Rust
Stars
53
Forks
16
Avg merge
4h 22m
Merged PRs (30d)
46

Description

## Why

The lexer stores byte offsets internally while diagnostic columns count Unicode scalar values. `front/lexer/src/cursor.rs` documents this invariant, but no tests verify it.

## Scope

- Test identifiers containing non-ASCII alphabetic characters.
- Test lookahead and tokenization when multi-byte characters appear before operators and literals.
- Trigger a lexer diagnostic after non-ASCII text and assert the human column rather than the byte offset.
- Cover a newline after UTF-8 input to verify `line_start` is reset correctly.
- Do not introduce Unicode normalization or change identifier policy.

## Likely files

- `front/lexer/src/cursor.rs`
- `front/lexer/src/ident.rs`
- A new lexer integration test

## Completion criteria

- [ ] Multi-byte UTF-8 input never panics or slices at an invalid boundary.
- [ ] Identifier tokens preserve their original spelling.
- [ ] Diagnostic line and column assertions distinguish characters from bytes.
- [ ] `cargo test -p lexer --locked` passes.

Contributor guide

Open the contributing guide

Research direction

Read the UTF-8 offset invariant in front/lexer/src/cursor.rs and identifier handling in front/lexer/src/ident.rs, then inspect the existing lexer test layout. Add integration coverage for non-ASCII identifiers, lookahead, operators and literals, diagnostics after UTF-8 text, and newline line_start handling. Done means the listed assertions pass without invalid-boundary panics; run cargo test -p lexer --locked.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.