python / python/cpython

Memory errors during parsing may trigger a debug abort

Ouverte
#156,108 5 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

interpreter-core topic-parser type-crash
Langage dominant
Python
Étoiles
77.2k
Forks
35.9k
Métriques de PR en attente

Description

Crash report

What happened?
import os
import sys
import _testcapi

source = b'''class E:
    def __iter__(self):
        yield 1
        yield from ()
def f(a, b=0, *args, **kwargs): pass
f(*E(), **{&a": 3})'''

for counter in range(200, 1200):
    print("Trying:", counter)
    sys.stdout.flush()
    _testcapi.set_nomemory(counter, counter + 1)
    try:
        _testcapi.run_compilestringflags(source, b"/missing.py", _testcapi.Py_file_input)
    except Exception:
        pass
    _testcapi.remove_mem_hooks()

For me, I get (ymmv):

Trying: 200
Trying: 201
Trying: 202
...
Trying: 340
Trying: 341
python3: Objects/call.c:342: _PyObject_Call: Assertion `!_PyErr_Occurred(tstate)' failed.

This is related to these issues/prs:

A fix was put in for this type of issue already in _PyPegen_raise_error_known_location:
https://github.com/python/cpython/blob/e8158d1a02d6e5e974d4dbc0633b6a3ad960d7a3/Parser/pegen_errors.c#L276-L279

and _PyPegen_raise_error:
https://github.com/python/cpython/blob/e8158d1a02d6e5e974d4dbc0633b6a3ad960d7a3/Parser/pegen_errors.c#L185-L187

But in this case, PyErr_Occurred() == True but p->error_indicator=0, because the allocation failure happened in an area of the code that could backtrack, it seems like the error_indicator isn't set.

Two code patterns that hit both sites are:

class E:
    def __iter__(self):
        yield 1
        yield from ()
def f(a, b=0, *args, **kwargs): pass
f(*E(), **{&a": 3})

and

class E:
    def __iter__(self):
        yield 1
        yield from ()
def f(a, b=0, *args, **kwargs): pass
with f(*E())
    pass

(The with here is the key)

So, to my mind, there are two questions:

  • Is it valid to get to either of these places with PyErr_Occurred() but not p->error_indicator?
  • If it is, then does the check just become checking PyErr_Occurred()?
  • If not, then the places where the parser checks for memory error and returns should probably always set the parser error_indicator. I guess technically a mis-predicted parsing branch could cause a MemoryError, and then backtrack and find the correct branch without triggering another MemoryError?? but istm that if you hit an allocation failure on a backtrackable branch, the only valid thing to do is to bail out.

LLDB info

LLDB 
(lldb) thread list
Process 2661 stopped
* thread #1: tid = 2661, 0x0000fffff692a848 libc.so.6`___lldb_unnamed_symbol_8a720 + 296, name = 'fuzz_python', stop reason = SIGABRT: sent by tkill system call (sender pid=2661, uid=0)

(lldb) bt all
* thread #1, name = 'fuzz_python', stop reason = SIGABRT: sent by tkill system call (sender pid=2661, uid=0)
  * frame #0: 0x0000fffff692a848 libc.so.6`___lldb_unnamed_symbol_8a720 + 296
    frame #1: 0x0000fffff68d7af8 libc.so.6`raise + 24
    frame #2: 0x0000fffff68c2630 libc.so.6`abort + 40
    frame #3: 0x0000fffff691dd84 libc.so.6`___lldb_unnamed_symbol_7dbe0 + 420
    frame #4: 0x0000fffff68d13ac libc.so.6`__assert_fail + 204
    frame #5: 0x0000aaaaaaee0834 fuzz_python`_PyObject_Call(tstate=0x0000aaaaab776358, callable=0x0000aaaaab6f4100, args=0x0000fffff4eb8f50, kwargs=0x0000000000000000) at call.c:342:5
    frame #6: 0x0000aaaaaad35d70 fuzz_python`PyObject_Call(callable=0x0000aaaaab6f4100, args=0x0000fffff4eb8f50, kwargs=0x0000000000000000) at call.c:373:12 [inlined]
    frame #7: 0x0000aaaaaad35d5c fuzz_python`PyErr_SetFromErrnoWithFilenameObjects(exc=0x0000aaaaab6f4100, filenameObject=0x0000fffff507eb60, filenameObject2=0x0000000000000000) at errors.c:900:13
    frame #8: 0x0000aaaaab0f5a7c fuzz_python`PyErr_SetFromErrnoWithFilenameObject(exc=<unavailable>, filenameObject=0x0000fffff507eb60) at errors.c:811:12 [inlined]
    frame #9: 0x0000aaaaab0f5a78 fuzz_python`Py_fopen(path=0x0000fffff507eb60, mode="rb") at fileutils.c:1817:9
    frame #10: 0x0000aaaaaad3ce30 fuzz_python`_PyErr_ProgramDecodedTextObject(filename=0x0000fffff507eb60, lineno=7, encoding=0x0000000000000000) at errors.c:2064:16
    frame #11: 0x0000aaaaab23e734 fuzz_python`_PyPegen_raise_error_known_location(p=0x0000fffff516a740, errtype=0x0000aaaaab6f6bd8, lineno=7, col_offset=12, end_lineno=7, end_col_offset=13, errmsg="Invalid star expression", va=<unavailable>) at pegen_errors.c:302:22
    frame #12: 0x0000aaaaab23e360 fuzz_python`_PyPegen_raise_error(p=0x0000fffff516a740, errtype=0x0000aaaaab6f6bd8, use_mark=0, errmsg="Invalid star expression") at pegen_errors.c:221:5
    frame #13: 0x0000aaaaab2fad34 fuzz_python`invalid_starred_expression_rule(p=0x0000fffff516a740) at parser.c:27408:20
    frame #14: 0x0000aaaaab2fa614 fuzz_python`starred_expression_rule(p=0x0000fffff516a740) at parser.c:18983:47
    frame #15: 0x0000aaaaab2d59a8 fuzz_python`_tmp_89_rule(p=0x0000fffff516a740) at parser.c:34278:39
    frame #16: 0x0000aaaaab2d58a0 fuzz_python`genexp_rule(p=0x0000fffff516a740) at parser.c:18464:18
    frame #17: 0x0000aaaaab2f3e68 fuzz_python`t_primary_raw(p=0x0000fffff516a740) at parser.c:20208:18
    frame #18: 0x0000aaaaab2f3400 fuzz_python`t_primary_rule(p=0x0000fffff516a740) at parser.c:20077:22
    frame #19: 0x0000aaaaab34a674 fuzz_python`single_subscript_attribute_target_rule(p=0x0000fffff516a740) at parser.c:19968:18
    frame #20: 0x0000aaaaab32773c fuzz_python`_tmp_11_rule(p=0x0000fffff516a740) at parser.c:29309:54
    frame #21: 0x0000aaaaab2b9878 fuzz_python`assignment_rule(p=0x0000fffff516a740) at parser.c:2196:18
    frame #22: 0x0000aaaaab2b97cc fuzz_python`simple_stmt_rule(p=0x0000fffff516a740) at parser.c:1620:31
    frame #23: 0x0000aaaaab2ae8c0 fuzz_python`simple_stmts_rule(p=0x0000fffff516a740) at parser.c:1515:18
    frame #24: 0x0000aaaaab2aafc0 fuzz_python`statement_rule(p=0x0000fffff516a740) at parser.c:1297:34
    frame #25: 0x0000aaaaab2aad28 fuzz_python`_loop1_2_rule(p=0x0000fffff516a740) at parser.c:28730:30
    frame #26: 0x0000aaaaab2aab60 fuzz_python`statements_rule(p=0x0000fffff516a740) at parser.c:1230:18
    frame #27: 0x0000aaaaab2a7578 fuzz_python`file_rule(p=0x0000fffff516a740) at parser.c:1032:18
    frame #28: 0x0000aaaaab2a7510 fuzz_python`_PyPegen_parse(p=0x0000fffff516a740) at parser.c:39618:18
    frame #29: 0x0000aaaaab22d3cc fuzz_python`_PyPegen_run_parser(p=0x0000fffff516a740) at pegen.c:1019:9
    frame #30: 0x0000aaaaab22f308 fuzz_python`_PyPegen_run_parser_from_string(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", start_rule=257, filename_ob=0x0000fffff507eb60, flags=0x0000000000000000, arena=0x0000fffff51772e0, module=<unavailable>) at pegen.c:1145:14
    frame #31: 0x0000aaaaab0f1c14 fuzz_python`_PyParser_ASTFromString(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", filename=0x0000fffff507eb60, mode=257, flags=0x0000000000000000, arena=0x0000fffff51772e0, module=0x0000000000000000) at peg_api.c:14:21 [inlined]
    frame #32: 0x0000aaaaab0f1bd8 fuzz_python`_Py_CompileString(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", filename=0x0000fffff507eb60, start=257, flags=0x0000000000000000, optimize=-1, module=0x0000000000000000) at pythonrun.c:1578:11
    frame #33: 0x0000aaaaab0f1ef0 fuzz_python`Py_CompileStringObject(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", filename=0x0000fffff507eb60, start=257, flags=0x0000000000000000, optimize=-1) at pythonrun.c:1561:12 [inlined]
    frame #34: 0x0000aaaaab0f1ed4 fuzz_python`Py_CompileStringExFlags(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", filename_str="<fuzz>", start=257, flags=0x0000000000000000, optimize=-1) at pythonrun.c:1608:10
    frame #35: 0x0000aaaaaad15054 fuzz_python`main(argc=<unavailable>, argv=<unavailable>) at fuzz_python.c:576:30
    frame #36: 0x0000fffff68c2d94 libc.so.6`___lldb_unnamed_symbol_22d20 + 116
    frame #37: 0x0000fffff68c2ef8 libc.so.6`__libc_start_main + 152
    frame #38: 0x0000aaaaaad06370 fuzz_python`_start + 48

(lldb) register read
General Purpose Registers:
        x0 = 0x0000000000000000
        x1 = 0x0000000000000a65
        x2 = 0x0000000000000006
        x3 = 0x0000000000000022
        x4 = 0xffffffffffffffff
        x5 = 0x0000000000000000
        x6 = 0x0000aaaaaab9e165  
        x7 = 0x0000fffff69f9750  
        x8 = 0x0000000000000083
        x9 = 0x5210160bf182cd92
       x10 = 0x0000000000000021
       x11 = 0x0000fffff6a66618  
       x12 = 0x0000000000000480
       x13 = 0x0000fffff7f03740
       x14 = 0x0000fffff51036f0
       x15 = 0x0000000000000001
       x16 = 0x0000000000000001
       x17 = 0x0000fffff693a36c  libc.so.6`__libc_free
       x18 = 0x0000000000000002
       x19 = 0x0000000000000a65
       x20 = 0x0000fffff7f03020
       x21 = 0x000000000000000b
       x22 = 0x0000000000000006
       x23 = 0x0000000000000000
       x24 = 0x000000000000009d
       x25 = 0x0000ffffffff7a38
       x26 = 0x0000fffff69f9740  
       x27 = 0x0000000000000002
       x28 = 0x0000fffff51a0000
        fp = 0x0000ffffffff7900
        lr = 0x0000fffff692a834  libc.so.6`___lldb_unnamed_symbol_8a720 + 276
        sp = 0x0000ffffffff78f0
        pc = 0x0000fffff692a848  libc.so.6`___lldb_unnamed_symbol_8a720 + 296
      cpsr = 0x60001000


(lldb) memory region $sp
[0x0000fffffffdd000-0x0001000000000000) rw-
CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.16.0a0 (heads/main-dirty:20e6c2f, Aug 19 2026, 15:22:30) [Clang 22.1.8 ]

Guide de contribution

Ouvrir le guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Piste de recherche

Commencez par reproduire l’échec avec la _testcapi memory-failure loop fournie, puis lisez Parser/pegen_errors.c et l’assertion dans Objects/call.c. Suivez la gestion de error_indicator par l’analyseur pour les deux modèles de code fournis et comparez-la aux chemins d’erreur liés. C’est terminé lorsque le reproducer déclenche ou signale l’échec d’allocation sans provoquer le debug abort.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
c, python
Domaine
compilers
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
Calme
Clarté
Plutôt claire
Accessibilité débutants
45/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.