python / python/cpython

`os.sendfile` leaks header buffer exports on validation errors

Offen
#156,287 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

extension-modules OS-freebsd OS-mac type-bug
Vorherrschende Sprache
Python
Sterne
77.2k
Forks
35.9k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

Bug report

Bug description:

On macOS (and as far as I know FreeBSD), os.sendfile takes (among others) two headers and trailers parameters. Calling iov_setup acquires Py_buffer exports for headers, but several validation steps that come after can error and return before iov_cleanup is ever called.

The setup is done here:

https://github.com/python/cpython/blob/f74cdf80a120649e4c353430da8cbd1305c00993/Modules/posixmodule.c#L12532-L12550

The early returns after that never call iov_cleanup:

https://github.com/python/cpython/blob/f74cdf80a120649e4c353430da8cbd1305c00993/Modules/posixmodule.c#L12551-L12584

iov_cleanup is only called for trailers and headers after the syscall:

https://github.com/python/cpython/blob/f74cdf80a120649e4c353430da8cbd1305c00993/Modules/posixmodule.c#L12602-L12605

In practice, this can happen if trailers is not valid for example. The function (legitimately) raises a TypeError, but:

  • There's a leak: iov_setup allocated an iovec array and a Py_buffer array with PyMem_New. (A call to iov_cleanup would free them, but it is not called.)
  • The input is left in an inconsistent state: PyObject_GetBuffer exported each header buffer. Without PyBuffer_Release, that export stays for the lifetime of the process, so for example a bytearray in headers can no longer be resized after the failure.

The same happens on macOS when the header-size overflow check returns after iov_setup, and when iov_setup for trailers fails after headers were already exported.

Reproducer

import os
import tempfile

header = bytearray(b"header")
with tempfile.TemporaryFile() as src, tempfile.TemporaryFile() as dst:
    try:
        os.sendfile(
            dst.fileno(), src.fileno(), 0, 0,
            headers=[header], trailers=object(),
        )
    except TypeError as exc:
        print(exc)  # sendfile() trailers must be a sequence

header.append(0)
# BufferError: Existing exports of data: object cannot be re-sized

The leak is properly reported by ASan/LSan:

=================================================================
==81993==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 80 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100fbbc00 in iov_setup posixmodule.c:12077
    #2 0x000100fae4d0 in os_sendfile_impl posixmodule.c:12454
    #3 0x000100fae4d0 in os_sendfile posixmodule.c.h:8233
    #4 0x000100a6e23c in _PyObject_VectorcallTstate pycore_call.h:144
    #5 0x000100a6e23c in PyObject_Vectorcall call.c:327
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100ddb36c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796
    #17 0x000100f7b978 in pymain_main main.c:826

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100fbbbc0 in iov_setup posixmodule.c:12071
    #2 0x000100fae4d0 in os_sendfile_impl posixmodule.c:12454
    #3 0x000100fae4d0 in os_sendfile posixmodule.c.h:8233
    #4 0x000100a6e23c in _PyObject_VectorcallTstate pycore_call.h:144
    #5 0x000100a6e23c in PyObject_Vectorcall call.c:327
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100ddb36c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796
    #17 0x000100f7b978 in pymain_main main.c:826

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100bd3684 in _PyObject_MallocWithType pycore_object_alloc.h:46
    #2 0x000100bd3684 in _PyType_AllocNoTrack typeobject.c:2523
    #3 0x000100bd345c in PyType_GenericAlloc typeobject.c:2554
    #4 0x000100bdd884 in type_call typeobject.c:2467
    #5 0x000100a6cd90 in _PyObject_MakeTpCall call.c:242
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100de4dc0 in _PyEval_EvalFrameDefault generated_cases.c.h:1846
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796

Indirect leak of 39 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100a5c9a4 in _PyBytes_FromSize bytesobject.c:121
    #2 0x000100a5c9a4 in _PyBytes_Resize bytesobject.c:3356
    #3 0x000100a3620c in bytearray_resize_lock_held bytearrayobject.c:280
    #4 0x000100a37b50 in PyByteArray_Resize bytearrayobject.c:299
    #5 0x000100a37b50 in bytearray___init___impl bytearrayobject.c:1021
    #6 0x000100a37b50 in bytearray___init__ bytearrayobject.c.h:102
    #7 0x000100bddab4 in type_call typeobject.c:2479
    #8 0x000100a6cd90 in _PyObject_MakeTpCall call.c:242
    #9 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #10 0x000100de4dc0 in _PyEval_EvalFrameDefault generated_cases.c.h:1846
    #11 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #12 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #13 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #14 0x000100f19ea4 in run_mod pythonrun.c:1472
    #15 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #16 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573

SUMMARY: AddressSanitizer: 199 byte(s) leaked in 4 allocation(s).
CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs
  • gh-156288

Beitragsleitfaden

Beitragsleitfaden öffnen

Erste Schritte

  1. Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
  3. Forke das Repository und arbeite in einem Branch.
  4. Öffne einen Pull Request, der die Issue-Nummer nennt.

Rechercherichtung

Beginne in Modules/posixmodule.c bei iov_setup und os_sendfile und folge den im Bericht identifizierten Validierungspfaden. Verwende den bereitgestellten macOS-Reproducer, um zu überprüfen, dass Validierungsfehler Ressourcen von Header und Trailer freigeben, Puffer veränderbar lassen und keine Allokationen leaken; der verknüpfte PR gh-156288 zeigt an, dass die Arbeit bereits im Gange ist.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
c, python
Bereich
operating-systems
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Veraltet
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.