python / python/cpython

`os.sendfile` leaks header buffer exports on validation errors

Open
#156,287 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

extension-modules OS-freebsd OS-mac type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

On macOS (and as far as I know FreeBSD), os.sendfile takes (among others) two headers and trailers parameters. Calling iov_setup acquires Py_buffer exports for headers, but several validation steps that come after can error and return before iov_cleanup is ever called.

The setup is done here:

https://github.com/python/cpython/blob/f74cdf80a120649e4c353430da8cbd1305c00993/Modules/posixmodule.c#L12532-L12550

The early returns after that never call iov_cleanup:

https://github.com/python/cpython/blob/f74cdf80a120649e4c353430da8cbd1305c00993/Modules/posixmodule.c#L12551-L12584

iov_cleanup is only called for trailers and headers after the syscall:

https://github.com/python/cpython/blob/f74cdf80a120649e4c353430da8cbd1305c00993/Modules/posixmodule.c#L12602-L12605

In practice, this can happen if trailers is not valid for example. The function (legitimately) raises a TypeError, but:

  • There's a leak: iov_setup allocated an iovec array and a Py_buffer array with PyMem_New. (A call to iov_cleanup would free them, but it is not called.)
  • The input is left in an inconsistent state: PyObject_GetBuffer exported each header buffer. Without PyBuffer_Release, that export stays for the lifetime of the process, so for example a bytearray in headers can no longer be resized after the failure.

The same happens on macOS when the header-size overflow check returns after iov_setup, and when iov_setup for trailers fails after headers were already exported.

Reproducer

import os
import tempfile

header = bytearray(b"header")
with tempfile.TemporaryFile() as src, tempfile.TemporaryFile() as dst:
    try:
        os.sendfile(
            dst.fileno(), src.fileno(), 0, 0,
            headers=[header], trailers=object(),
        )
    except TypeError as exc:
        print(exc)  # sendfile() trailers must be a sequence

header.append(0)
# BufferError: Existing exports of data: object cannot be re-sized

The leak is properly reported by ASan/LSan:

=================================================================
==81993==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 80 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100fbbc00 in iov_setup posixmodule.c:12077
    #2 0x000100fae4d0 in os_sendfile_impl posixmodule.c:12454
    #3 0x000100fae4d0 in os_sendfile posixmodule.c.h:8233
    #4 0x000100a6e23c in _PyObject_VectorcallTstate pycore_call.h:144
    #5 0x000100a6e23c in PyObject_Vectorcall call.c:327
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100ddb36c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796
    #17 0x000100f7b978 in pymain_main main.c:826

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100fbbbc0 in iov_setup posixmodule.c:12071
    #2 0x000100fae4d0 in os_sendfile_impl posixmodule.c:12454
    #3 0x000100fae4d0 in os_sendfile posixmodule.c.h:8233
    #4 0x000100a6e23c in _PyObject_VectorcallTstate pycore_call.h:144
    #5 0x000100a6e23c in PyObject_Vectorcall call.c:327
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100ddb36c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796
    #17 0x000100f7b978 in pymain_main main.c:826

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100bd3684 in _PyObject_MallocWithType pycore_object_alloc.h:46
    #2 0x000100bd3684 in _PyType_AllocNoTrack typeobject.c:2523
    #3 0x000100bd345c in PyType_GenericAlloc typeobject.c:2554
    #4 0x000100bdd884 in type_call typeobject.c:2467
    #5 0x000100a6cd90 in _PyObject_MakeTpCall call.c:242
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100de4dc0 in _PyEval_EvalFrameDefault generated_cases.c.h:1846
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796

Indirect leak of 39 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100a5c9a4 in _PyBytes_FromSize bytesobject.c:121
    #2 0x000100a5c9a4 in _PyBytes_Resize bytesobject.c:3356
    #3 0x000100a3620c in bytearray_resize_lock_held bytearrayobject.c:280
    #4 0x000100a37b50 in PyByteArray_Resize bytearrayobject.c:299
    #5 0x000100a37b50 in bytearray___init___impl bytearrayobject.c:1021
    #6 0x000100a37b50 in bytearray___init__ bytearrayobject.c.h:102
    #7 0x000100bddab4 in type_call typeobject.c:2479
    #8 0x000100a6cd90 in _PyObject_MakeTpCall call.c:242
    #9 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #10 0x000100de4dc0 in _PyEval_EvalFrameDefault generated_cases.c.h:1846
    #11 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #12 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #13 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #14 0x000100f19ea4 in run_mod pythonrun.c:1472
    #15 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #16 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573

SUMMARY: AddressSanitizer: 199 byte(s) leaked in 4 allocation(s).
CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs
  • gh-156288

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in Modules/posixmodule.c at iov_setup and os_sendfile, following the validation paths identified in the report. Use the provided macOS reproducer to verify that validation failures release header and trailer resources, leave buffers resizable, and do not leak allocations; linked PR gh-156288 indicates work is already underway.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.