[Bug]: tokenize emits a non-existent new line token for empty input
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 155
- Avg merge
- 9h 42m
- Merged PRs (30d)
- 36
Description
Describe the bug
When using token.generate_tokens() on an empty string, a NL token is returned when no such token exists for CPython. The return token also spans a non-existent length, which caused stubgen-pyx to crash. Only effects "", # ... and "\n" were fine. This caused stubgen-pyx to crash on an otherwise empty file.
Likely occurs in GraalPy's re-implementation of _tokenize.TokenizerIter
Operating system
Linux
CPU architecture
x86_64
GraalPy version
GraalPy 3.13.14 (GraalVM CE Native 25.3.4.1)
JDK version
openjdk 25.0.4 2026-07-21 LTS (Temurin-25.0.4+7)
Context configuration
N/A
Steps to reproduce
# repro.py
import io, tokenize
for tok in tokenize.generate_tokens(io.StringIO("").readline):
print(tok)
$ graalpy repro.py
TokenInfo(type=63 (NL), string='', start=(1, 0), end=(1, 1), line='')
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')
$ python3.13 repro.py # and python3.14, identical
TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')
Native version
import io, _tokenize
for t in _tokenize.TokenizerIter(io.StringIO("").readline, extra_tokens=True):
print(t)
#GraalPy 3.13.14: (63, '', (1, 0), (1, 1), '') ← phantom NL
# (0, '', (2, 0), (2, 0), '')
#CPython 3.13: (0, '', (1, 0), (1, 0), '')
Expected behavior
Should match the same output as CPython 3.13, and not emit non-existent spans.
Stack trace
N/A
Additional context
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the repro.py example with GraalPy and CPython, then trace the GraalPy implementation of _tokenize.TokenizerIter used by tokenize.generate_tokens. The fix is done when empty input produces only the CPython-compatible ENDMARKER with matching positions, while the existing comment and newline cases remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100