Keyword typo suggestions differ between file and string tokenizers
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 77.2k
- 派生
- 35.9k
- PR 合并指标
- PR 指标待抓取
描述
Bug report
Bug description:
Keyword-typo SyntaxError suggestions depend on whether CPython processes the same source using the file tokenizer or the string tokenizer.
Create t.py containing:
a=b=c=d=e=f=g=h=i=0
def fn():
retrun True
Running the file directly:
$ python t.py
uses the file-tokenizer path and produces the helpful diagnostic:
File ".../t.py", line 3
retrun True
^^^^^^
SyntaxError: invalid syntax. Did you mean 'return'?
Running the same file as a module:
$ python -m t
uses the string-tokenizer path and produces:
File ".../t.py", line 3
retrun True
^^^^
SyntaxError: invalid syntax
The same discrepancy is visible in the default PyREPL-based REPL.
Pasting the complete block as one input:
a=b=c=d=e=f=g=h=i=0
def fn():
retrun True
produces the generic error:
File "<python-input-0>", line 3
retrun True
^^^^
SyntaxError: invalid syntax
Entering and submitting the assignment separately, followed by the function:
a=b=c=d=e=f=g=h=i=0
def fn():
retrun True
produces:
File "<python-input-2>", line 2
retrun True
^^^^^^
SyntaxError: invalid syntax. Did you mean 'return'?
I would expect the same source code to produce the same keyword suggestion regardless of whether it is:
- executed as a file with
python t.py; - loaded as a module with
python -m t; - pasted into PyREPL as one block; or
- entered into PyREPL statement by statement.
This appears related to the keyword-typo suggestions added in #132449.
traceback.TracebackException._find_keyword_typos() treats the two source paths differently:
- When the source is loaded from the filename, tokens outside the reported error line are skipped before consuming the token-search budget.
- When the source is included directly in the
SyntaxErrormetadata, all non-keywordNAMEtokens in the extracted source fragment consume that budget.
The search is limited to ten such tokens. In this example, a through i account for nine tokens and fn is the tenth, so the string-source path stops before examining retrun. The file-source path skips those earlier lines and finds the typo.
One possible approach would be to prioritize tokens on the reported error line for both source paths, and only then search the surrounding source fragment. That would preserve the existing work limit while avoiding this tokenizer-dependent result.
CPython versions tested on:
CPython main branch, 3.15, 3.14
Operating systems tested on:
macOS
Linked PRs
- gh-156087
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
从 traceback.TracebackException._find_keyword_typos() 开始,对比其 filename 和 SyntaxError 元数据路径,以及此处描述的十个 token 预算。复现上面的四种执行模式;当同一源代码在文件执行、模块加载和两种 PyREPL 输入形式下产生相同的关键字建议时,即表示完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- compilers
- Issue 类型
- 缺陷
- 难度
- 3/5
- 预计耗时
- 1-2 天
- 活跃度
- 停滞
- 描述清晰度
- 描述清楚
- 新手友好度
- 25/100