python / python/cpython

Memory errors during parsing may trigger a debug abort

Open
#156,108 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

interpreter-core topic-parser type-crash
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR metrics pending

Description

Crash report

What happened?
import os
import sys
import _testcapi

source = b'''class E:
    def __iter__(self):
        yield 1
        yield from ()
def f(a, b=0, *args, **kwargs): pass
f(*E(), **{&a": 3})'''

for counter in range(200, 1200):
    print("Trying:", counter)
    sys.stdout.flush()
    _testcapi.set_nomemory(counter, counter + 1)
    try:
        _testcapi.run_compilestringflags(source, b"/missing.py", _testcapi.Py_file_input)
    except Exception:
        pass
    _testcapi.remove_mem_hooks()

For me, I get (ymmv):

Trying: 200
Trying: 201
Trying: 202
...
Trying: 340
Trying: 341
python3: Objects/call.c:342: _PyObject_Call: Assertion `!_PyErr_Occurred(tstate)' failed.

This is related to these issues/prs:

A fix was put in for this type of issue already in _PyPegen_raise_error_known_location:
https://github.com/python/cpython/blob/e8158d1a02d6e5e974d4dbc0633b6a3ad960d7a3/Parser/pegen_errors.c#L276-L279

and _PyPegen_raise_error:
https://github.com/python/cpython/blob/e8158d1a02d6e5e974d4dbc0633b6a3ad960d7a3/Parser/pegen_errors.c#L185-L187

But in this case, PyErr_Occurred() == True but p->error_indicator=0, because the allocation failure happened in an area of the code that could backtrack, it seems like the error_indicator isn't set.

Two code patterns that hit both sites are:

class E:
    def __iter__(self):
        yield 1
        yield from ()
def f(a, b=0, *args, **kwargs): pass
f(*E(), **{&a": 3})

and

class E:
    def __iter__(self):
        yield 1
        yield from ()
def f(a, b=0, *args, **kwargs): pass
with f(*E())
    pass

(The with here is the key)

So, to my mind, there are two questions:

  • Is it valid to get to either of these places with PyErr_Occurred() but not p->error_indicator?
  • If it is, then does the check just become checking PyErr_Occurred()?
  • If not, then the places where the parser checks for memory error and returns should probably always set the parser error_indicator. I guess technically a mis-predicted parsing branch could cause a MemoryError, and then backtrack and find the correct branch without triggering another MemoryError?? but istm that if you hit an allocation failure on a backtrackable branch, the only valid thing to do is to bail out.

LLDB info

LLDB 
(lldb) thread list
Process 2661 stopped
* thread #1: tid = 2661, 0x0000fffff692a848 libc.so.6`___lldb_unnamed_symbol_8a720 + 296, name = 'fuzz_python', stop reason = SIGABRT: sent by tkill system call (sender pid=2661, uid=0)

(lldb) bt all
* thread #1, name = 'fuzz_python', stop reason = SIGABRT: sent by tkill system call (sender pid=2661, uid=0)
  * frame #0: 0x0000fffff692a848 libc.so.6`___lldb_unnamed_symbol_8a720 + 296
    frame #1: 0x0000fffff68d7af8 libc.so.6`raise + 24
    frame #2: 0x0000fffff68c2630 libc.so.6`abort + 40
    frame #3: 0x0000fffff691dd84 libc.so.6`___lldb_unnamed_symbol_7dbe0 + 420
    frame #4: 0x0000fffff68d13ac libc.so.6`__assert_fail + 204
    frame #5: 0x0000aaaaaaee0834 fuzz_python`_PyObject_Call(tstate=0x0000aaaaab776358, callable=0x0000aaaaab6f4100, args=0x0000fffff4eb8f50, kwargs=0x0000000000000000) at call.c:342:5
    frame #6: 0x0000aaaaaad35d70 fuzz_python`PyObject_Call(callable=0x0000aaaaab6f4100, args=0x0000fffff4eb8f50, kwargs=0x0000000000000000) at call.c:373:12 [inlined]
    frame #7: 0x0000aaaaaad35d5c fuzz_python`PyErr_SetFromErrnoWithFilenameObjects(exc=0x0000aaaaab6f4100, filenameObject=0x0000fffff507eb60, filenameObject2=0x0000000000000000) at errors.c:900:13
    frame #8: 0x0000aaaaab0f5a7c fuzz_python`PyErr_SetFromErrnoWithFilenameObject(exc=<unavailable>, filenameObject=0x0000fffff507eb60) at errors.c:811:12 [inlined]
    frame #9: 0x0000aaaaab0f5a78 fuzz_python`Py_fopen(path=0x0000fffff507eb60, mode="rb") at fileutils.c:1817:9
    frame #10: 0x0000aaaaaad3ce30 fuzz_python`_PyErr_ProgramDecodedTextObject(filename=0x0000fffff507eb60, lineno=7, encoding=0x0000000000000000) at errors.c:2064:16
    frame #11: 0x0000aaaaab23e734 fuzz_python`_PyPegen_raise_error_known_location(p=0x0000fffff516a740, errtype=0x0000aaaaab6f6bd8, lineno=7, col_offset=12, end_lineno=7, end_col_offset=13, errmsg="Invalid star expression", va=<unavailable>) at pegen_errors.c:302:22
    frame #12: 0x0000aaaaab23e360 fuzz_python`_PyPegen_raise_error(p=0x0000fffff516a740, errtype=0x0000aaaaab6f6bd8, use_mark=0, errmsg="Invalid star expression") at pegen_errors.c:221:5
    frame #13: 0x0000aaaaab2fad34 fuzz_python`invalid_starred_expression_rule(p=0x0000fffff516a740) at parser.c:27408:20
    frame #14: 0x0000aaaaab2fa614 fuzz_python`starred_expression_rule(p=0x0000fffff516a740) at parser.c:18983:47
    frame #15: 0x0000aaaaab2d59a8 fuzz_python`_tmp_89_rule(p=0x0000fffff516a740) at parser.c:34278:39
    frame #16: 0x0000aaaaab2d58a0 fuzz_python`genexp_rule(p=0x0000fffff516a740) at parser.c:18464:18
    frame #17: 0x0000aaaaab2f3e68 fuzz_python`t_primary_raw(p=0x0000fffff516a740) at parser.c:20208:18
    frame #18: 0x0000aaaaab2f3400 fuzz_python`t_primary_rule(p=0x0000fffff516a740) at parser.c:20077:22
    frame #19: 0x0000aaaaab34a674 fuzz_python`single_subscript_attribute_target_rule(p=0x0000fffff516a740) at parser.c:19968:18
    frame #20: 0x0000aaaaab32773c fuzz_python`_tmp_11_rule(p=0x0000fffff516a740) at parser.c:29309:54
    frame #21: 0x0000aaaaab2b9878 fuzz_python`assignment_rule(p=0x0000fffff516a740) at parser.c:2196:18
    frame #22: 0x0000aaaaab2b97cc fuzz_python`simple_stmt_rule(p=0x0000fffff516a740) at parser.c:1620:31
    frame #23: 0x0000aaaaab2ae8c0 fuzz_python`simple_stmts_rule(p=0x0000fffff516a740) at parser.c:1515:18
    frame #24: 0x0000aaaaab2aafc0 fuzz_python`statement_rule(p=0x0000fffff516a740) at parser.c:1297:34
    frame #25: 0x0000aaaaab2aad28 fuzz_python`_loop1_2_rule(p=0x0000fffff516a740) at parser.c:28730:30
    frame #26: 0x0000aaaaab2aab60 fuzz_python`statements_rule(p=0x0000fffff516a740) at parser.c:1230:18
    frame #27: 0x0000aaaaab2a7578 fuzz_python`file_rule(p=0x0000fffff516a740) at parser.c:1032:18
    frame #28: 0x0000aaaaab2a7510 fuzz_python`_PyPegen_parse(p=0x0000fffff516a740) at parser.c:39618:18
    frame #29: 0x0000aaaaab22d3cc fuzz_python`_PyPegen_run_parser(p=0x0000fffff516a740) at pegen.c:1019:9
    frame #30: 0x0000aaaaab22f308 fuzz_python`_PyPegen_run_parser_from_string(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", start_rule=257, filename_ob=0x0000fffff507eb60, flags=0x0000000000000000, arena=0x0000fffff51772e0, module=<unavailable>) at pegen.c:1145:14
    frame #31: 0x0000aaaaab0f1c14 fuzz_python`_PyParser_ASTFromString(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", filename=0x0000fffff507eb60, mode=257, flags=0x0000000000000000, arena=0x0000fffff51772e0, module=0x0000000000000000) at peg_api.c:14:21 [inlined]
    frame #32: 0x0000aaaaab0f1bd8 fuzz_python`_Py_CompileString(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", filename=0x0000fffff507eb60, start=257, flags=0x0000000000000000, optimize=-1, module=0x0000000000000000) at pythonrun.c:1578:11
    frame #33: 0x0000aaaaab0f1ef0 fuzz_python`Py_CompileStringObject(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", filename=0x0000fffff507eb60, start=257, flags=0x0000000000000000, optimize=-1) at pythonrun.c:1561:12 [inlined]
    frame #34: 0x0000aaaaab0f1ed4 fuzz_python`Py_CompileStringExFlags(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", filename_str="<fuzz>", start=257, flags=0x0000000000000000, optimize=-1) at pythonrun.c:1608:10
    frame #35: 0x0000aaaaaad15054 fuzz_python`main(argc=<unavailable>, argv=<unavailable>) at fuzz_python.c:576:30
    frame #36: 0x0000fffff68c2d94 libc.so.6`___lldb_unnamed_symbol_22d20 + 116
    frame #37: 0x0000fffff68c2ef8 libc.so.6`__libc_start_main + 152
    frame #38: 0x0000aaaaaad06370 fuzz_python`_start + 48

(lldb) register read
General Purpose Registers:
        x0 = 0x0000000000000000
        x1 = 0x0000000000000a65
        x2 = 0x0000000000000006
        x3 = 0x0000000000000022
        x4 = 0xffffffffffffffff
        x5 = 0x0000000000000000
        x6 = 0x0000aaaaaab9e165  
        x7 = 0x0000fffff69f9750  
        x8 = 0x0000000000000083
        x9 = 0x5210160bf182cd92
       x10 = 0x0000000000000021
       x11 = 0x0000fffff6a66618  
       x12 = 0x0000000000000480
       x13 = 0x0000fffff7f03740
       x14 = 0x0000fffff51036f0
       x15 = 0x0000000000000001
       x16 = 0x0000000000000001
       x17 = 0x0000fffff693a36c  libc.so.6`__libc_free
       x18 = 0x0000000000000002
       x19 = 0x0000000000000a65
       x20 = 0x0000fffff7f03020
       x21 = 0x000000000000000b
       x22 = 0x0000000000000006
       x23 = 0x0000000000000000
       x24 = 0x000000000000009d
       x25 = 0x0000ffffffff7a38
       x26 = 0x0000fffff69f9740  
       x27 = 0x0000000000000002
       x28 = 0x0000fffff51a0000
        fp = 0x0000ffffffff7900
        lr = 0x0000fffff692a834  libc.so.6`___lldb_unnamed_symbol_8a720 + 276
        sp = 0x0000ffffffff78f0
        pc = 0x0000fffff692a848  libc.so.6`___lldb_unnamed_symbol_8a720 + 296
      cpsr = 0x60001000


(lldb) memory region $sp
[0x0000fffffffdd000-0x0001000000000000) rw-
CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.16.0a0 (heads/main-dirty:20e6c2f, Aug 19 2026, 15:22:30) [Clang 22.1.8 ]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the provided _testcapi memory-failure loop, then read Parser/pegen_errors.c and the assertion in Objects/call.c. Trace the parser error_indicator handling for the two supplied code patterns and compare it with the linked error paths. Done means the reproducer raises or reports the allocation failure without triggering the debug abort.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.