Move the syntax error generation rules out of Grammar/python.gram into a separate grammar file
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 77.2k
- 派生
- 35.9k
- PR 合并指标
- PR 指标待抓取
描述
Feature or enhancement
Grammar/python.gram contains not only the Python grammar, but also the rules for generating specialized syntax error messages (the invalid_* rules and the alternatives referencing them) -- about a third of the file. They allow producing better error messages, but they are not part of the Python grammar:
- The full grammar specification in the documentation includes
python.gramand has to hide the error rules with fragile regular expressions in a custom Pygments lexer (#.*invalid syntax.*up to the end of file, every alternative mentioninginvalid_*). - Alternative Python implementations which consume the grammar get the CPython-specific error machinery mixed in. Some rules exist only to work around this (e.g.
invalid_parameters_helperwas added in GH-24156 "so that alternative implementations written in statically-typed languages can use this grammar without having type errors in the way"). - The error rules obscure the grammar itself: it is hard to see which alternatives define the language and which only improve error reporting, and the order of the alternatives mixes both concerns.
I propose to split it in two files:
Grammar/python.gram-- the pure Python grammar, without a single reference to the error rules. By itself it produces a complete parser which parses exactly the same language and only emits generic error messages.Grammar/python_errors.gram-- the alternatives for generating specialized syntax error messages, together with the information about where they are inserted into the rules of the main grammar.
The pegen parser generator reads both files and merges them. The error alternatives are written (with their actions) in rule extensions:
raise_stmt (extend):
| a='raise' b='from' {
RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "did you forget an expression between 'raise' and 'from'?") }
| 'raise' expression a='from' {
RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "did you forget an expression after 'from'?") }
The position of an inserted alternative is deduced automatically: it is inserted before the leftmost alternative of the base rule which can succeed by matching a proper prefix of the code matched by the inserted alternative (such alternative would shadow the specialized error if it was tried first), or after all alternatives if there is no such alternative. Alternatives which must preempt a base alternative which reports an error itself rather than failing (because it contains forced tokens or rules like block) are written before ..., which stands for the alternatives of the base rule. Several rules can be extended at once ((kwarg_or_starred | kwarg_or_double_starred) (extend):), and a rule can be extended several times. The inserted alternatives are only used in the second parsing pass, like the alternatives referencing invalid_* rules; only a dozen error rules which are shared, memoized or referenced from other alternatives remain defined as named rules.
The generated parser is not changed: the pure split reproduces Parser/parser.c byte for byte (except the generated-from comment), and the whole change has equivalent behavior, with only the order and grouping of the error alternatives changed. This was verified by a differential test which ran both parsers on a corpus of 873 invalid inputs covering every error message in the grammar (identical exception types, messages, locations) and on 157 stdlib files (identical AST, including attributes).
Linked PRs
- gh-153197
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
首先阅读 Grammar/python.gram 和提议的 Grammar/python_errors.gram 结构,然后检查 pegen parser generator 如何将它们组合起来。比较生成的 Parser/parser.c,以及针对无效输入和 stdlib 文件所描述的差分测试结果。完成的标准是:拆分后仍能保留 parser 的输出和行为,同时分离错误 alternatives。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- compilers
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 描述清楚
- 新手友好度
- 25/100