python / python/cpython

Incorrect SyntaxError attributes when raised by `compile`

未关闭
#127,927 11 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

interpreter-core type-bug
主要语言
Python
星标
77.2k
派生
36k
PR 合并指标
PR 指标待抓取

描述

Bug report

Bug description:

This is probably a problem with compile, and I will demonstrate it through ast.parse. Consider the following code in a file f.py.

import ast

code = "x y"
ast.parse(code, filename='f.py')

This correctly raises a SyntaxError with lineno = 1 and offset = 3 since the error is at the beginning of y in the code. However, the traceback points into the file f.py itself, instead of the string code. Notice that it points to the third column of the first line of f.py.

  File "f.py", line 1
    import ast
      ^
SyntaxError: invalid syntax

This does not happen for any other string given as filename to ast.parse. In general, the traceback seems to point with the "correct" lineno and offset to the code in the file itself instead of code given to ast.parse. However, the problem is slightly more nuanced. If the offset in the string given to ast.parse is more than the length of the corresponding line in f.py, the offset attribute of the SyntaxError raised is incorrectly set to the length of the corresponding line in f.py plus two. Consider the following code in f.py.

import ast

code = "                                     x y"
ast.parse(code, filename='f.py')

This raises the following exception.

  File "f.py", line 1
    import ast
              ^
IndentationError: unexpected indent

On closer inspection, it turns out that the offset of the exception raised is 12 (note that the length of "import ast" is 10) instead of the correct 37 which is observed when filename is set to anything other than f.py. However, something more interesting happens when the number of lines in the code string is more than the number of lines in the file f.py.

import ast

code = "\n\n\n\nx y"
ast.parse(code, filename='f.py')

The problem goes away!

  File "f.py", line 5
    x y
      ^
SyntaxError: invalid syntax

We even see the correct offset.

CPython versions tested on:

3.11, 3.12, 3.13

Operating systems tested on:

Linux

(Edit) Incorrect text attribute

There is something more going on here. The following issue can be seen in 3.11 and 3.12 but not in 3.13. Consider the following f.py.

import ast

with open('temp.py') as f:
    code = f.read()
ast.parse(code, filename='temp.py')

temp.py is as follows.

"""                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             """; x """                                                                                                                                                         """; foobar

This leads to the following traceback.

  File "temp.py", line 1
    foobar
          ^
SyntaxError: invalid syntax

Besides the fact that the offset is incorrectly set to 8 (instead of 840), the text is incorrectly set to foobar (on 3.13, the entire file contents of temp.py are present in text). This problem goes away if filename is set to anything else. The problem also goes away if the contents of temp.py are shorter!

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

首先使用 ast.parse 和 compile 重现这些示例,比较 filename 与源文件匹配和不匹配时的 SyntaxError 属性。跟踪文件名匹配时 traceback、offset 和 text 的填充方式。当在所示案例中,lineno、offset、text 和 traceback 始终一致地描述已解析的字符串,并且这些示例具有回归测试覆盖时,即可视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
compilers
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
38/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。