tokenize: `e.offset` on `IndentationError` is not set correctly
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 77.2k
- 派生
- 35.9k
- PR 合并指标
- PR 指标待抓取
描述
Bug report
Bug description:
import io, tokenize
input_s = """
def foo():
x = 1
x = "some_very_long_expression"
"""
try:
list(tokenize.generate_tokens(io.StringIO(input_s).readline))
except IndentationError as e:
print(f"{e.offset = }")
The line x = "some_very_long_expression" is indented with 3 spaces instead of 4, so we get an error.
We would expect the offset to be around 3, since that is where the issue is.
On python 3.12+, the offset is instead 35, equal to the length of the line + 1.
This appears to be caused by the refactor of tokenize done in 3.12 for PEP 701
Potentially related to https://github.com/python/cpython/issues/84515
Funnily enough, the exact same thing seems to have happened around 3.8 https://github.com/python/cpython/issues/75676
Tested versions:
| Version | e.offset |
|---|---|
| 3.11.15 | 3 |
| 3.12.13 | 35 |
| 3.13.13 | 35 |
| 3.14.4 | 35 |
| 3.15.0b1 | 35 |
Testing command:
nix-shell -p python311 --run "python --version && python tmp.py"
(and replace 311 by 312, 313, etc)
CPython versions tested on:
3.15
Operating systems tested on:
Linux
Linked PRs
- gh-154281
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
从 tokenize.generate_tokens 开始,在列出的 Python 版本中重现所提供的 IndentationError 示例。跟踪 tokenizer 如何构造 IndentationError,并将 offset 与 Python 3.11 的行为进行比较。当缩进错误的行报告的 offset 接近 3,并且该示例具有回归覆盖时,即表示完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- compilers
- Issue 类型
- 缺陷
- 难度
- 3/5
- 预计耗时
- 1-2 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100