python / python/cpython

`os.sendfile` leaks header buffer exports on validation errors

Ouverte
#156,287 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

extension-modules OS-freebsd OS-mac type-bug
Langage dominant
Python
Étoiles
77.2k
Forks
35.9k
Métriques de merge des PR
Métriques de PR en attente

Description

Bug report

Bug description:

On macOS (and as far as I know FreeBSD), os.sendfile takes (among others) two headers and trailers parameters. Calling iov_setup acquires Py_buffer exports for headers, but several validation steps that come after can error and return before iov_cleanup is ever called.

The setup is done here:

https://github.com/python/cpython/blob/f74cdf80a120649e4c353430da8cbd1305c00993/Modules/posixmodule.c#L12532-L12550

The early returns after that never call iov_cleanup:

https://github.com/python/cpython/blob/f74cdf80a120649e4c353430da8cbd1305c00993/Modules/posixmodule.c#L12551-L12584

iov_cleanup is only called for trailers and headers after the syscall:

https://github.com/python/cpython/blob/f74cdf80a120649e4c353430da8cbd1305c00993/Modules/posixmodule.c#L12602-L12605

In practice, this can happen if trailers is not valid for example. The function (legitimately) raises a TypeError, but:

  • There's a leak: iov_setup allocated an iovec array and a Py_buffer array with PyMem_New. (A call to iov_cleanup would free them, but it is not called.)
  • The input is left in an inconsistent state: PyObject_GetBuffer exported each header buffer. Without PyBuffer_Release, that export stays for the lifetime of the process, so for example a bytearray in headers can no longer be resized after the failure.

The same happens on macOS when the header-size overflow check returns after iov_setup, and when iov_setup for trailers fails after headers were already exported.

Reproducer

import os
import tempfile

header = bytearray(b"header")
with tempfile.TemporaryFile() as src, tempfile.TemporaryFile() as dst:
    try:
        os.sendfile(
            dst.fileno(), src.fileno(), 0, 0,
            headers=[header], trailers=object(),
        )
    except TypeError as exc:
        print(exc)  # sendfile() trailers must be a sequence

header.append(0)
# BufferError: Existing exports of data: object cannot be re-sized

The leak is properly reported by ASan/LSan:

=================================================================
==81993==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 80 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100fbbc00 in iov_setup posixmodule.c:12077
    #2 0x000100fae4d0 in os_sendfile_impl posixmodule.c:12454
    #3 0x000100fae4d0 in os_sendfile posixmodule.c.h:8233
    #4 0x000100a6e23c in _PyObject_VectorcallTstate pycore_call.h:144
    #5 0x000100a6e23c in PyObject_Vectorcall call.c:327
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100ddb36c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796
    #17 0x000100f7b978 in pymain_main main.c:826

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100fbbbc0 in iov_setup posixmodule.c:12071
    #2 0x000100fae4d0 in os_sendfile_impl posixmodule.c:12454
    #3 0x000100fae4d0 in os_sendfile posixmodule.c.h:8233
    #4 0x000100a6e23c in _PyObject_VectorcallTstate pycore_call.h:144
    #5 0x000100a6e23c in PyObject_Vectorcall call.c:327
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100ddb36c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796
    #17 0x000100f7b978 in pymain_main main.c:826

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100bd3684 in _PyObject_MallocWithType pycore_object_alloc.h:46
    #2 0x000100bd3684 in _PyType_AllocNoTrack typeobject.c:2523
    #3 0x000100bd345c in PyType_GenericAlloc typeobject.c:2554
    #4 0x000100bdd884 in type_call typeobject.c:2467
    #5 0x000100a6cd90 in _PyObject_MakeTpCall call.c:242
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100de4dc0 in _PyEval_EvalFrameDefault generated_cases.c.h:1846
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796

Indirect leak of 39 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100a5c9a4 in _PyBytes_FromSize bytesobject.c:121
    #2 0x000100a5c9a4 in _PyBytes_Resize bytesobject.c:3356
    #3 0x000100a3620c in bytearray_resize_lock_held bytearrayobject.c:280
    #4 0x000100a37b50 in PyByteArray_Resize bytearrayobject.c:299
    #5 0x000100a37b50 in bytearray___init___impl bytearrayobject.c:1021
    #6 0x000100a37b50 in bytearray___init__ bytearrayobject.c.h:102
    #7 0x000100bddab4 in type_call typeobject.c:2479
    #8 0x000100a6cd90 in _PyObject_MakeTpCall call.c:242
    #9 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #10 0x000100de4dc0 in _PyEval_EvalFrameDefault generated_cases.c.h:1846
    #11 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #12 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #13 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #14 0x000100f19ea4 in run_mod pythonrun.c:1472
    #15 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #16 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573

SUMMARY: AddressSanitizer: 199 byte(s) leaked in 4 allocation(s).
CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs
  • gh-156288

Guide de contribution

Ouvrir le guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Piste de recherche

Commencez dans Modules/posixmodule.c, au niveau de iov_setup et os_sendfile, en suivant les chemins de validation identifiés dans le rapport. Utilisez le reproducer macOS fourni pour vérifier que les échecs de validation libèrent les ressources du header et du trailer, laissent les buffers redimensionnables et n’entraînent aucune fuite d’allocations ; la PR liée gh-156288 indique que le travail est déjà en cours.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
c, python
Domaine
operating-systems
Type d'issue
Bug
Difficulté
3/5
Temps estimé
1-2 jours
Activité
À l'abandon
Clarté
Clairement spécifiée
Accessibilité débutants
25/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.