python / python/cpython

`_csv.reader`: NULL deref via re-entrant iterator that reaches EOF with an open quoted field

Offen
#157,379 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

extension-modules type-bug
Vorherrschende Sprache
Python
Sterne
77.2k
Forks
35.9k
PR-Kennzahlen ausstehend

Beschreibung

Bug report

Bug description:

csv.reader() crashes with a NULL pointer dereference in parse_save_field()
when its input iterator re-enters the same reader and the outer call then hits
end of input while state == IN_QUOTED_FIELD.

This is a second, still open path of gh-145105 (the fix in PR #145106 added the
self->fields == NULL guard only after a successful PyIter_Next(), not in the
end-of-input branch).

Reproduction

import csv


class It:
    def __init__(self):
        self.n = 0
        self.reader = None

    def __iter__(self):
        return self

    def __next__(self):
        self.n += 1
        if self.n == 1:
            next(self.reader)  # re-enter the same reader
            raise StopIteration
        if self.n == 2:
            return '"x'  # inner call: opens a quoted field
        raise StopIteration


it = It()
r = csv.reader(it)
it.reader = r
print(next(r))

python repro.py segfaults.

Expected

A csv.Error (the same "iterator has already advanced the reader" raised by
the gh-145105 fix), or at worst a clean StopIteration. No crash.

Actual

Fatal Python error: Segmentation fault
...
  #5 0x... in parse_save_field .../Modules/_csv.c:713
  #6 0x... in Reader_iternext_lock_held .../Modules/_csv.c:978

Affected versions

build result
main (575fe3914f4, pydebug + ASAN + UBSAN) NULL deref in PyList_Append, Fatal Python error: Segmentation fault
3.15.0rc2+dev (e325fae3578) AddressSanitizer: SEGV on unknown address 0x000000000008 in _Py_TYPE_impl / PyList_Append
3.14.7 Segmentation fault (core dumped), exit 139
3.13.15+ Segmentation fault (core dumped), exit 139

Not a regression in main; the bug is shared by all maintained branches.

Cause

Reader_iternext_lock_held() (Modules/_csv.c). When the input iterator returns
NULL, the end-of-input branch saves a pending field:

lineobj = PyIter_Next(self->input_iter);
if (lineobj == NULL) {
    /* End of input OR exception */
    if (!PyErr_Occurred() && (self->field_len != 0 ||
                              self->state == IN_QUOTED_FIELD)) {
        if (self->dialect->strict)
            PyErr_SetString(module_state->error_obj,
                            "unexpected end of data");
        else if (parse_save_field(self) >= 0)   /* <- _csv.c:978 */
            break;
    }
    return NULL;
}
if (!PyUnicode_Check(lineobj)) { ... }
if (self->fields == NULL) {                 /* <- gh-145105 guard, too late */
    PyErr_SetString(module_state->error_obj,
                    "iterator has already advanced the reader");
    ...
}

A re-entrant next(reader) inside the input iterator completes successfully and
sets self->fields = NULL, but leaves self->state == IN_QUOTED_FIELD (the row
was returned through the end-of-input break path without resetting the state).
Control returns to the outer call, whose PyIter_Next() now gets
StopIteration. The outer call enters the lineobj == NULL branch and calls
parse_save_field(), which does PyList_Append(self->fields, field) with
self->fields == NULL.

Workaround

csv.reader(..., strict=True) raises _csv.Error: unexpected end of data
instead of crashing, because the strict branch returns before
parse_save_field().

Related

  • gh-145105 _csv.reader: NULL deref via re-entrant iterator (closed)
  • PR #145106 (main), #148404 (3.14), #148405 (3.13) - the incomplete fix
  • No issue or PR found for this end-of-input path

Build used for the sanitizer output

CC=clang CXX=clang++ LDFLAGS='-fuse-ld=lld' ./configure \
    --with-pydebug --with-address-sanitizer --with-undefined-behavior-sanitizer

Run as ASAN_OPTIONS=detect_leaks=0 ./python -X dev repro.py.

Full output on main (UBSan + ASan) 
Include/object.h:234:16: runtime error: member access within null pointer of type 'PyObject' (aka 'struct _object')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Include/object.h:234:16 
Fatal Python error: Segmentation fault

Current thread 0x00007fc6676fd400 [python] (most recent call first):
  File "/tmp/csvbug/repro.py", line 25 in <module>

Current thread's C stack trace (most recent call first):
  Binary file "./python", at +0xde4bdd [0x557ff6a8ebdd]
  Binary file "./python", at _Py_DumpStack+0x10c [0x557ff78c5f24]
  Binary file "./python", at +0x1c97ee7 [0x557ff7941ee7]
  Binary file "/usr/lib/libc.so.6", at +0x3e6f0 [0x7fc66743e6f0]
  Binary file "./python", at PyList_Append+0x51 [0x557ff708d969]
  Binary file "/home/boss/projects/oss/cpython/main/build/lib.linux-x86_64-3.16/_csv.cpython-316d-x86_64-linux-gnu.so", at +0x1ab06 [0x7fc66695eb06]
  Binary file "/home/boss/projects/oss/cpython/main/build/lib.linux-x86_64-3.16/_csv.cpython-316d-x86_64-linux-gnu.so", at +0x19bf2 [0x7fc66695dbf2]
  Binary file "./python", at +0x183c5ef [0x557ff74e65ef]
  Binary file "./python", at +0x14ac791 [0x557ff7156791]
  Binary file "./python", at +0x12f6fd6 [0x557ff6fa0fd6]
  Binary file "./python", at _Py_VectorCallInstrumentation_StackRefSteal+0x179 [0x557ff74f3be1]
  Binary file "./python", at _PyEval_EvalFrameDefault+0x5239a [0x557ff7549012]
  Binary file "./python", at PyEval_EvalCode+0x2d9 [0x557ff74f2931]
  Binary file "./python", at +0x1b9b43b [0x557ff784543b]
  Binary file "./python", at +0x1b9ab62 [0x557ff7844b62]
  Binary file "./python", at +0x1b95670 [0x557ff783f670]
  Binary file "./python", at +0x1b938b5 [0x557ff783d8b5]
  Binary file "./python", at +0x1b92e8d [0x557ff783ce8d]
  Binary file "./python", at +0x1c5cc10 [0x557ff7906c10]
  Binary file "./python", at Py_RunMain+0xe48 [0x557ff7905300]
  Binary file "./python", at +0x1c5c350 [0x557ff7906350]
  Binary file "./python", at Py_BytesMain+0xd5 [0x557ff79065dd]
  Binary file "/usr/lib/libc.so.6", at +0x27781 [0x7fc667427781]
  Binary file "/usr/lib/libc.so.6", at __libc_start_main+0x89 [0x7fc6674278b9]
  Binary file "./python", at _start+0x25 [0x557ff6a26a65]
AddressSanitizer:DEADLYSIGNAL
=================================================================
==637567==ERROR: AddressSanitizer: SEGV on unknown address 0x03e80009ba7f (pc 0x7fc66749a17c bp 0x7e56667f2c50 sp 0x7e56667f2c20 T0)
==637567==The signal is caused by a READ memory access.
    #0 0x7fc66749a17c  (/usr/lib/libc.so.6+0x9a17c) (BuildId: 503200d7fda94a5dc6058d7e0694e5d1dcb2e372)
    #1 0x7fc66743e5cf in raise (/usr/lib/libc.so.6+0x3e5cf) (BuildId: 503200d7fda94a5dc6058d7e0694e5d1dcb2e372)
    #2 0x7fc66743e6ef  (/usr/lib/libc.so.6+0x3e6ef) (BuildId: 503200d7fda94a5dc6058d7e0694e5d1dcb2e372)
    #3 0x557ff708d968 in _Py_TYPE_impl /home/boss/projects/oss/cpython/main/./Include/object.h:234:16
    #4 0x557ff708d968 in PyList_Append /home/boss/projects/oss/cpython/main/Objects/listobject.c:541:9
    #5 0x7fc66695eb05 in parse_save_field /home/boss/projects/oss/cpython/main/./Modules/_csv.c:713:9
    #6 0x7fc66695dbf1 in Reader_iternext_lock_held /home/boss/projects/oss/cpython/main/./Modules/_csv.c:978:26
    #7 0x7fc66695dbf1 in Reader_iternext /home/boss/projects/oss/cpython/main/./Modules/_csv.c:1028:14
    #8 0x557ff74e65ee in builtin_next /home/boss/projects/oss/cpython/main/Python/bltinmodule.c:1777:11
    #9 0x557ff7156790 in cfunction_vectorcall_FASTCALL /home/boss/projects/oss/cpython/main/Objects/methodobject.c:449:24
    #10 0x557ff6fa0fd5 in _PyObject_VectorcallTstate /home/boss/projects/oss/cpython/main/./Include/internal/pycore_call.h:144:11
    #11 0x557ff74f3be0 in _Py_VectorCallInstrumentation_StackRefSteal /home/boss/projects/oss/cpython/main/Python/ceval.c:770:11
    #12 0x557ff7549011 in _PyEval_EvalFrameDefault /home/boss/projects/oss/cpython/main/Python/generated_cases.c.h:1906:35
    #13 0x557ff74f2930 in _PyEval_Vector /home/boss/projects/oss/cpython/main/Python/ceval.c:2176:12
    #14 0x557ff74f2930 in PyEval_EvalCode /home/boss/projects/oss/cpython/main/Python/ceval.c:681:21
    #15 0x557ff784543a in run_eval_code_obj /home/boss/projects/oss/cpython/main/Python/pythonrun.c:1406:12
    #16 0x557ff7844b61 in run_mod /home/boss/projects/oss/cpython/main/Python/pythonrun.c:1509:19
    #17 0x557ff783f66f in _PyRun_File /home/boss/projects/oss/cpython/main/Python/pythonrun.c:1332:15
    #18 0x557ff783d8b4 in _PyRun_SimpleFile /home/boss/projects/oss/cpython/main/Python/pythonrun.c:544:15
    #19 0x557ff783ce8c in _PyRun_AnyFile /home/boss/projects/oss/cpython/main/Python/pythonrun.c:92:18
    #20 0x557ff7906c0f in pymain_run_file_obj /home/boss/projects/oss/cpython/main/Modules/main.c:478:24
    #21 0x557ff7906c0f in pymain_run_file /home/boss/projects/oss/cpython/main/Modules/main.c:494:20
    #22 0x557ff79052ff in pymain_run_python /home/boss/projects/oss/cpython/main/Modules/main.c:804:21
    #23 0x557ff79052ff in Py_RunMain /home/boss/projects/oss/cpython/main/Modules/main.c:891:5
    #24 0x557ff790634f in pymain_main /home/boss/projects/oss/cpython/main/Modules/main.c:921:12
    #25 0x557ff79065dc in Py_BytesMain /home/boss/projects/oss/cpython/main/Modules/main.c:945:12
    #26 0x7fc667427780  (/usr/lib/libc.so.6+0x27780) (BuildId: 503200d7fda94a5dc6058d7e0694e5d1dcb2e372)
    #27 0x7fc6674278b8 in __libc_start_main (/usr/lib/libc.so.6+0x278b8) (BuildId: 503200d7fda94a5dc6058d7e0694e5d1dcb2e372)
    #28 0x557ff6a26a64 in _start (/home/boss/projects/oss/cpython/main/python+0xd7ca64) (BuildId: f312134e13fb614412b5e6b12b00d0f08bdbd276)

==637567==Register values:
rax = 0x0000000000000000  rbx = 0x000000000009ba7f  rcx = 0x00007fc66749a17c  rdx = 0x000000000000000b  
rdi = 0x000000000009ba7f  rsi = 0x000000000009ba7f  rbp = 0x00007e56667f2c50  rsp = 0x00007e56667f2c20  
 r8 = 0x0000557ff82bd528   r9 = 0x0000000000000038  r10 = 0x0000000000000000  r11 = 0x0000000000000246  
r12 = 0x00007c86668042e0  r13 = 0x00000000ffffffff  r14 = 0x000000000000000b  r15 = 0x0000000000000008  
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/usr/lib/libc.so.6+0x9a17c) (BuildId: 503200d7fda94a5dc6058d7e0694e5d1dcb2e372) 
==637567==ABORTING
Full output on 3.15 (UBSan + ASan) 
Include/object.h:234:16: runtime error: member access within null pointer of type 'PyObject' (aka 'struct _object')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Include/object.h:234:16 
AddressSanitizer:DEADLYSIGNAL
=================================================================
==637570==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x55e3b4990149 bp 0x7ffedac5fbe0 sp 0x7ffedac5fbc0 T0)
==637570==The signal is caused by a READ memory access.
==637570==Hint: address points to the zero page.
    #0 0x55e3b4990149 in _Py_TYPE_impl /home/boss/projects/oss/cpython/3.15/./Include/object.h:234:16
    #1 0x55e3b4990149 in PyList_Append /home/boss/projects/oss/cpython/3.15/Objects/listobject.c:541:9
    #2 0x7f67751ea725 in parse_save_field /home/boss/projects/oss/cpython/3.15/./Modules/_csv.c:688:9
    #3 0x7f67751e9811 in Reader_iternext_lock_held /home/boss/projects/oss/cpython/3.15/./Modules/_csv.c:953:26
    #4 0x7f67751e9811 in Reader_iternext /home/boss/projects/oss/cpython/3.15/./Modules/_csv.c:1003:14
    #5 0x55e3b4de936e in builtin_next /home/boss/projects/oss/cpython/3.15/Python/bltinmodule.c:1776:11
    #6 0x55e3b4a5759f in cfunction_vectorcall_FASTCALL /home/boss/projects/oss/cpython/3.15/Objects/methodobject.c:449:24
    #7 0x55e3b48a8415 in _PyObject_VectorcallTstate /home/boss/projects/oss/cpython/3.15/./Include/internal/pycore_call.h:144:11
    #8 0x55e3b4df6ad0 in _Py_VectorCallInstrumentation_StackRefSteal /home/boss/projects/oss/cpython/3.15/Python/ceval.c:777:11
    #9 0x55e3b4e3a16b in _PyEval_EvalFrameDefault /home/boss/projects/oss/cpython/3.15/Python/generated_cases.c.h:1846:35
    #10 0x55e3b4df5820 in _PyEval_Vector /home/boss/projects/oss/cpython/3.15/Python/ceval.c:2153:12
    #11 0x55e3b4df5820 in PyEval_EvalCode /home/boss/projects/oss/cpython/3.15/Python/ceval.c:688:21
    #12 0x55e3b51110ca in run_eval_code_obj /home/boss/projects/oss/cpython/3.15/Python/pythonrun.c:1398:12
    #13 0x55e3b51107f1 in run_mod /home/boss/projects/oss/cpython/3.15/Python/pythonrun.c:1501:19
    #14 0x55e3b510b25f in _PyRun_File /home/boss/projects/oss/cpython/3.15/Python/pythonrun.c:1324:15
    #15 0x55e3b510931e in _PyRun_SimpleFile /home/boss/projects/oss/cpython/3.15/Python/pythonrun.c:536:15
    #16 0x55e3b51088fc in _PyRun_AnyFile /home/boss/projects/oss/cpython/3.15/Python/pythonrun.c:84:18
    #17 0x55e3b51d0779 in pymain_run_file_obj /home/boss/projects/oss/cpython/3.15/Modules/main.c:437:24
    #18 0x55e3b51d0779 in pymain_run_file /home/boss/projects/oss/cpython/3.15/Modules/main.c:458:20
    #19 0x55e3b51ce8ad in pymain_run_python /home/boss/projects/oss/cpython/3.15/Modules/main.c:750:21
    #20 0x55e3b51ce8ad in Py_RunMain /home/boss/projects/oss/cpython/3.15/Modules/main.c:837:5
    #21 0x55e3b51cf97f in pymain_main /home/boss/projects/oss/cpython/3.15/Modules/main.c:867:12
    #22 0x55e3b51cfc0c in Py_BytesMain /home/boss/projects/oss/cpython/3.15/Modules/main.c:891:12
    #23 0x7f6775c27780  (/usr/lib/libc.so.6+0x27780) (BuildId: 503200d7fda94a5dc6058d7e0694e5d1dcb2e372)
    #24 0x7f6775c278b8 in __libc_start_main (/usr/lib/libc.so.6+0x278b8) (BuildId: 503200d7fda94a5dc6058d7e0694e5d1dcb2e372)
    #25 0x55e3b433ce64 in _start (/home/boss/projects/oss/cpython/3.15/python+0xd79e64) (BuildId: e127fbc7ed2e5451fe39515b06306c80828a344f)

==637570==Register values:
rax = 0x0000000000000001  rbx = 0x0000000000000000  rcx = 0x00007ffedac5f868  rdx = 0x0000000000000000  
rdi = 0x00007ffedac5f800  rsi = 0x0000000000000000  rbp = 0x00007ffedac5fbe0  rsp = 0x00007ffedac5fbc0  
 r8 = 0x00007f6775200760   r9 = 0x000055e3b5c06a40  r10 = 0x000055e3b5bf9200  r11 = 0x0000000000000202  
r12 = 0x00007c27750001a0  r13 = 0x00000000ffffffff  r14 = 0x000055e3b5b89c30  r15 = 0x0000000000000008  
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/boss/projects/oss/cpython/3.15/./Include/object.h:234:16 in _Py_TYPE_impl
==637570==ABORTING
3.13 / 3.14 output 
$ /home/boss/.local/bin/python3.13 repro.py
Segmentation fault (core dumped)   # exit 139
$ python3 repro.py
Segmentation fault (core dumped)   # exit 139
CPython versions tested on:

CPython main branch, 3.16, 3.15, 3.14, 3.13

Operating systems tested on:

Linux

Linked PRs
  • gh-157511

Beitragsleitfaden

Beitragsleitfaden öffnen

Erste Schritte

  1. Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
  3. Forke das Repository und arbeite in einem Branch.
  4. Öffne einen Pull Request, der die Issue-Nummer nennt.

Rechercherichtung

Beginnen Sie in Modules/_csv.c bei Reader_iternext_lock_held(), parse_save_field() und der bestehenden self->fields-Schutzprüfung, die für gh-145105 hinzugefügt wurde. Reproduzieren Sie den Fall eines wiedereintrittsfähigen Iterators mit dem bereitgestellten Skript und fügen Sie anschließend eine Regressionstestabdeckung hinzu, die zeigt, dass EOF nach dem Wiedereintritt entweder csv.Error auslöst oder sauber stoppt, statt NULL zu dereferenzieren.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
c, python
Bereich
backend
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Aktiv
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
52/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.