wavefnd / wavefnd/Wave

Preserve byte semantics for non-ASCII string hex escapes

Open
#599 0 comments 0 reactions 0 assignees View on GitHub
bug help wanted needs testing
Dominant language
Rust
Stars
53
Forks
16
Avg merge
4h 22m
Merged PRs (30d)
46

Description

The lexer documents `\\xNN` string escapes as accepting the full byte range `00` through `FF`.

After parsing the two hex digits as `u8`, however, the string scanner currently does:

`string_literal.push(value as char)`

The literal payload is a Rust `String`, so values above ASCII are encoded as Unicode scalar values and then represented in UTF-8.

For example, the requested byte value `0xFF` becomes the character U+00FF, whose UTF-8 representation is `0xC3 0xBF`, rather than preserving one byte `0xFF`.

The same representation also permits `\\x00` to insert an embedded NUL into the lexer `String`. Wave's public `str` helpers are null-terminated, so the exact contract for embedded zero bytes needs to be made explicit across the lexer, parser, and code generator.

Character literals take a different path: the parsed value is checked against the unsigned 8-bit range, so this issue is specifically about string-literal payload representation and lowering.

Code evidence:

- `front/lexer/src/literals.rs` — `Lexer::string`
- `front/lexer/src/literals.rs` — `Lexer::char_literal`
- `std/string/*` — null-terminated `str` operations

Acceptance:

- [ ] Define whether `\\xNN` in a Wave string denotes an exact byte or a Unicode code point.
- [ ] Make the implementation match the documented `00` through `FF` contract.
- [ ] Cover `\\x00`, `\\x7F`, `\\x80`, and `\\xFF` through the complete frontend/backend path.
- [ ] Ensure string literal length and emitted bytes are tested, not only lexer token text.
- [ ] Keep ordinary UTF-8 source characters and existing ASCII escapes working.
- [ ] If embedded NUL is unsupported by the current `str` ABI, reject it explicitly with a diagnostic rather than silently creating a truncated runtime string.

Contributor guide

Open the contributing guide

Research direction

Start with Lexer::string and Lexer::char_literal in front/lexer/src/literals.rs, then trace string lowering into the backend and the null-terminated operations under std/string/*. Run the existing frontend and backend tests before changing behavior. Done means the byte contract is explicit, edge cases from 00 through FF are covered end to end, ordinary UTF-8 and ASCII escapes still work, and embedded NUL is either supported or diagnosed explicitly.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.