Preserve byte semantics for non-ASCII string hex escapes
- Dominant language
- Rust
- Stars
- 53
- Forks
- 16
- Avg merge
- 4h 22m
- Merged PRs (30d)
- 46
Description
The lexer documents `\\xNN` string escapes as accepting the full byte range `00` through `FF`.
After parsing the two hex digits as `u8`, however, the string scanner currently does:
`string_literal.push(value as char)`
The literal payload is a Rust `String`, so values above ASCII are encoded as Unicode scalar values and then represented in UTF-8.
For example, the requested byte value `0xFF` becomes the character U+00FF, whose UTF-8 representation is `0xC3 0xBF`, rather than preserving one byte `0xFF`.
The same representation also permits `\\x00` to insert an embedded NUL into the lexer `String`. Wave's public `str` helpers are null-terminated, so the exact contract for embedded zero bytes needs to be made explicit across the lexer, parser, and code generator.
Character literals take a different path: the parsed value is checked against the unsigned 8-bit range, so this issue is specifically about string-literal payload representation and lowering.
Code evidence:
- `front/lexer/src/literals.rs` — `Lexer::string`
- `front/lexer/src/literals.rs` — `Lexer::char_literal`
- `std/string/*` — null-terminated `str` operations
Acceptance:
- [ ] Define whether `\\xNN` in a Wave string denotes an exact byte or a Unicode code point.
- [ ] Make the implementation match the documented `00` through `FF` contract.
- [ ] Cover `\\x00`, `\\x7F`, `\\x80`, and `\\xFF` through the complete frontend/backend path.
- [ ] Ensure string literal length and emitted bytes are tested, not only lexer token text.
- [ ] Keep ordinary UTF-8 source characters and existing ASCII escapes working.
- [ ] If embedded NUL is unsupported by the current `str` ABI, reject it explicitly with a diagnostic rather than silently creating a truncated runtime string.
Contributor guide
Research direction
Start with Lexer::string and Lexer::char_literal in front/lexer/src/literals.rs, then trace string lowering into the backend and the null-terminated operations under std/string/*. Run the existing frontend and backend tests before changing behavior. Done means the byte contract is explicit, edge cases from 00 through FF are covered end to end, ordinary UTF-8 and ASCII escapes still work, and embedded NUL is either supported or diagnosed explicitly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100