Two conformance gaps: \0 followed by a digit is accepted, and underscores are silently dropped from numbers
- Dominant language
- Cython
- Stars
- 179
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
`pyjson5` 2.0.1 (current release) accepts two classes of input that the JSON5 spec forbids and that both reference implementations — the JS `json5` package (2.2.3) and the pure-Python `json5` package (dpranke, 0.15.0) — reject.
Found by differential fuzzing: the same 3,000 seeded inputs to pyjson5 and the JS reference, comparing accept/reject and parsed values. Both classes reproduce deterministically.
## Class A: `\0` followed by a digit
The spec (following ECMAScript 5.1) allows `\0` **only when not followed by a decimal digit** — the rule exists to keep legacy octal escapes out. pyjson5's `_get_escape_sequence` handles `\\1`–`\\9` correctly (they raise), but the `b'0'` branch returns NUL unconditionally, with no lookahead:
```pyx
elif c0 == b'0':
return 0x0000 # src/_decoder.pyx — missing: lookahead ∉ DecimalDigit
```
```python
import pyjson5
pyjson5.loads('"\\01"') # '\x001' — should raise (JS json5 and dpranke both reject)
pyjson5.loads('"\\012"') # '\x0012' — should raise
pyjson5.loads('"\\00"') # '\x000' — should raise
pyjson5.loads('"\\0"') # '\x00' — correct, legal
```
## Class B: underscores silently dropped from numbers
JSON5 has no numeric separators (that's ES2021, not ES5.1). pyjson5's number scanners consume and discard `_` in three places (integer/hex, fraction, and exponent loops in `src/_decoder.pyx`, each with an `elif c0 == b'_'` skip), so:
```python
pyjson5.loads('1_0') # 10 — should raise
pyjson5.loads('1__0') # 10 — should raise
pyjson5.loads('1_') # 1 — should raise
pyjson5.loads('0x_1') # 1 — should raise
pyjson5.loads('.5_0') # 0.5 — should raise
pyjson5.loads('1e1_0') # 10000000000.0 — should raise
```
I checked the README, CHANGELOG, and issue tracker — neither behavior is documented or reported, and no test asserts the underscore behavior. If numeric separators are an intentional extension, a doc line would close this; as-is it looks accidental, especially `1_` → `1` and `.5_0` → `0.5`.
## Why these survived
The official `json5-tests` suite (which this repo runs via `third-party/json5-tests`) has no test for `\0`-followed-by-digit and none for underscore separators — the strings/ directory has five escape tests, none covering the `\0` lookahead. Both gaps live exactly where the conformance suite is blind. Adding cases there would protect every implementation that runs the suite, not just this one.
## Impact
Modest but real: cross-parser config confusion. A polyglot stack that validates JSON5 config with a strict parser and consumes it with pyjson5 (or the reverse) reads different documents from the same bytes.
## Disclosure
Found by AI agents (differential fuzzing harness) working under my direction; minimization, root-causing, and this report were reviewed and run by me. Happy to share the harness or turn either class into a PR with tests if you'd like.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/_decoder.pyx, focusing on _get_escape_sequence and the integer, fraction, and exponent number scanners described in the report. Review the relevant cases in third-party/json5-tests, then add coverage for \0 followed by digits and numeric underscores. Done means forbidden inputs are rejected while a standalone \0 remains accepted, with the conformance tests passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100