code-yeongyu / code-yeongyu/c11-compiler-zig-omo
[Sisyphus] phase-2 / zcc / Wave 1 — Lexer
- Dominant language
- C
- Stars
- 3
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Scope
Implement a complete C11 lexer that produces a token stream covering the full C11 token set.
## Deliverables
- `phase2/zcc/src/lex.zig` — lexer implementation.
- `phase2/zcc/include/zcc/lex.h` (or Zig-equivalent public interface) — token type definitions and lexer API.
## Token Coverage (must handle)
- Preprocessing numbers (`pp-number`).
- All C11 keywords and identifiers.
- String literals: UTF-8, UTF-16 (`u"..."`), UTF-32 (`U"..."`), wide (`L"..."`).
- Character literals with escape sequences.
- All operators and punctuators.
- C11-specific keywords: `_Atomic`, `_Generic`, `_Noreturn`, `_Static_assert`, `_Thread_local`, `_Alignof`, `_Alignas`.
## Success Criteria
1. Lexer tokenizes every `.c` file under `phase1/c11-ref/*.c` with **zero unknown tokens**.
2. Token stream is deterministic and reproducible across runs.
3. Error messages include source location (file, line, column).
## Dependencies on Prior Waves
- **Wave 0** — build.zig and driver entry must exist to compile and run lexer tests.
## Suggested Role Assignment
`lexer-engineer`
## Test Corpus Expectations
- **Unit tests** (given/when/then / AAA pattern): tokenize individual constructs (keywords, literals, operators) and assert expected token kinds/lexemes.
- **Corpus tests**: run lexer over all `phase1/c11-ref/*.c` source files; compare token-kind sequence against a reference (e.g., `clang -Xclang -dump-tokens` or a pre-approved golden file).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the existing structure under phase2/zcc and inspect phase2/zcc/src/lex.zig plus phase2/zcc/include/zcc/lex.h, or the Zig-equivalent interface. After Wave 0 provides build.zig and the driver, run the lexer unit tests and the phase1/c11-ref/*.c corpus. Done means full C11 token coverage, deterministic token streams, zero unknown tokens, and source locations in errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, zig
- Domain
- compilers
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100