code-yeongyu / code-yeongyu/c11-compiler-zig-omo

[Sisyphus] phase-2 / zcc / Wave 1 — Lexer

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
3
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Scope
Implement a complete C11 lexer that produces a token stream covering the full C11 token set.

## Deliverables
- `phase2/zcc/src/lex.zig` — lexer implementation.
- `phase2/zcc/include/zcc/lex.h` (or Zig-equivalent public interface) — token type definitions and lexer API.

## Token Coverage (must handle)
- Preprocessing numbers (`pp-number`).
- All C11 keywords and identifiers.
- String literals: UTF-8, UTF-16 (`u"..."`), UTF-32 (`U"..."`), wide (`L"..."`).
- Character literals with escape sequences.
- All operators and punctuators.
- C11-specific keywords: `_Atomic`, `_Generic`, `_Noreturn`, `_Static_assert`, `_Thread_local`, `_Alignof`, `_Alignas`.

## Success Criteria
1. Lexer tokenizes every `.c` file under `phase1/c11-ref/*.c` with **zero unknown tokens**.
2. Token stream is deterministic and reproducible across runs.
3. Error messages include source location (file, line, column).

## Dependencies on Prior Waves
- **Wave 0** — build.zig and driver entry must exist to compile and run lexer tests.

## Suggested Role Assignment
`lexer-engineer`

## Test Corpus Expectations
- **Unit tests** (given/when/then / AAA pattern): tokenize individual constructs (keywords, literals, operators) and assert expected token kinds/lexemes.
- **Corpus tests**: run lexer over all `phase1/c11-ref/*.c` source files; compare token-kind sequence against a reference (e.g., `clang -Xclang -dump-tokens` or a pre-approved golden file).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the existing structure under phase2/zcc and inspect phase2/zcc/src/lex.zig plus phase2/zcc/include/zcc/lex.h, or the Zig-equivalent interface. After Wave 0 provides build.zig and the driver, run the lexer unit tests and the phase1/c11-ref/*.c corpus. Done means full C11 token coverage, deterministic token streams, zero unknown tokens, and source locations in errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, zig
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.