Specialized syntax error messages are not produced in all positions where the invalid construct can occur

Feature or enhancement

Several rules for generating specialized syntax error messages are attached to a grammar rule in which the invalid construct can only occur at some of its possible positions. When the same construct occurs at a deeper position, the parser falls back to a generic SyntaxError: invalid syntax.

invalid_factor catches a unary operator followed by not. It is an alternative of term, so it is only tried where a term starts. But its construct occurs wherever a factor is expected -- also as the right operand of *, /, %, //, @ and **:

>>> + not x
SyntaxError: 'not' after an operator must be parenthesized
>>> 1 * + not x
SyntaxError: invalid syntax
>>> 2 ** + not x
SyntaxError: invalid syntax

invalid_arithmetic catches a binary arithmetic operator followed by not. It is an alternative of shift_expr, so it misses the construct in the right operand of << and >> (a sum position):

>>> 1 + not x
SyntaxError: 'not' after an operator must be parenthesized
>>> 1 << 2 + not x
SyntaxError: invalid syntax

The fix is to attach each error rule to the rule matched at all positions where its construct can occur:

• move invalid_factor from term to factor;
• move invalid_arithmetic from shift_expr to sum.

After this change all the examples above produce the specialized error messages, and all previously produced messages are unchanged.

The placement of invalid_factor and invalid_arithmetic dates from their introduction in bpo-24612 (GH-28170), which attached both symmetrically to the rule containing the binary operators, one level above the operand rule where the construct can actually occur.

There are a few related gaps which require modifying the error rules themselves rather than moving them (1 << not x, lambda x=: 0, def f[T=]: pass, x = 1 + *y); they are left for separate issues.

Linked PRs

• gh-153175
• gh-153193

Start with the grammar rules named in the issue: invalid_factor, invalid_arithmetic, term, factor, shift_expr, and sum. Move each error rule to the specified operand rule, then check the listed interactive examples to confirm specialized messages appear and existing messages remain unchanged.