python / python/cpython

Move the syntax error generation rules out of Grammar/python.gram into a separate grammar file

未關閉
#153,195 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

interpreter-core topic-parser type-feature
主要語言
Python
星號
77.2k
分支
36k
PR 合併指標
PR 指標待擷取

描述

Feature or enhancement

Grammar/python.gram contains not only the Python grammar, but also the rules for generating specialized syntax error messages (the invalid_* rules and the alternatives referencing them) -- about a third of the file. They allow producing better error messages, but they are not part of the Python grammar:

  • The full grammar specification in the documentation includes python.gram and has to hide the error rules with fragile regular expressions in a custom Pygments lexer (#.*invalid syntax.* up to the end of file, every alternative mentioning invalid_*).
  • Alternative Python implementations which consume the grammar get the CPython-specific error machinery mixed in. Some rules exist only to work around this (e.g. invalid_parameters_helper was added in GH-24156 "so that alternative implementations written in statically-typed languages can use this grammar without having type errors in the way").
  • The error rules obscure the grammar itself: it is hard to see which alternatives define the language and which only improve error reporting, and the order of the alternatives mixes both concerns.

I propose to split it in two files:

  • Grammar/python.gram -- the pure Python grammar, without a single reference to the error rules. By itself it produces a complete parser which parses exactly the same language and only emits generic error messages.
  • Grammar/python_errors.gram -- the alternatives for generating specialized syntax error messages, together with the information about where they are inserted into the rules of the main grammar.

The pegen parser generator reads both files and merges them. The error alternatives are written (with their actions) in rule extensions:

raise_stmt (extend):
    | a='raise' b='from' {
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "did you forget an expression between 'raise' and 'from'?") }
    | 'raise' expression a='from' {
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "did you forget an expression after 'from'?") }

The position of an inserted alternative is deduced automatically: it is inserted before the leftmost alternative of the base rule which can succeed by matching a proper prefix of the code matched by the inserted alternative (such alternative would shadow the specialized error if it was tried first), or after all alternatives if there is no such alternative. Alternatives which must preempt a base alternative which reports an error itself rather than failing (because it contains forced tokens or rules like block) are written before ..., which stands for the alternatives of the base rule. Several rules can be extended at once ((kwarg_or_starred | kwarg_or_double_starred) (extend):), and a rule can be extended several times. The inserted alternatives are only used in the second parsing pass, like the alternatives referencing invalid_* rules; only a dozen error rules which are shared, memoized or referenced from other alternatives remain defined as named rules.

The generated parser is not changed: the pure split reproduces Parser/parser.c byte for byte (except the generated-from comment), and the whole change has equivalent behavior, with only the order and grouping of the error alternatives changed. This was verified by a differential test which ran both parsers on a corpus of 873 invalid inputs covering every error message in the grammar (identical exception types, messages, locations) and on 157 stdlib files (identical AST, including attributes).

Linked PRs
  • gh-153197

貢獻指南

開啟貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

研究方向

先閱讀 Grammar/python.gram 和提議的 Grammar/python_errors.gram 結構,接著檢查 pegen parser generator 如何將它們組合起來。比較產生的 Parser/parser.c,以及針對無效輸入和 stdlib 檔案所描述的差分測試結果。完成的標準是:拆分後仍能保留 parser 的輸出和行為,同時分離錯誤 alternatives。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
compilers
Issue 類型
功能
難度
5/5
預估耗時
一週以上
活躍度
停滯
描述清晰度
描述清楚
新手友好度
25/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。