python / python/cpython

`os.sendfile` leaks header buffer exports on validation errors

未关闭
#156,287 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

extension-modules OS-freebsd OS-mac type-bug
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

Bug report

Bug description:

On macOS (and as far as I know FreeBSD), os.sendfile takes (among others) two headers and trailers parameters. Calling iov_setup acquires Py_buffer exports for headers, but several validation steps that come after can error and return before iov_cleanup is ever called.

The setup is done here:

https://github.com/python/cpython/blob/f74cdf80a120649e4c353430da8cbd1305c00993/Modules/posixmodule.c#L12532-L12550

The early returns after that never call iov_cleanup:

https://github.com/python/cpython/blob/f74cdf80a120649e4c353430da8cbd1305c00993/Modules/posixmodule.c#L12551-L12584

iov_cleanup is only called for trailers and headers after the syscall:

https://github.com/python/cpython/blob/f74cdf80a120649e4c353430da8cbd1305c00993/Modules/posixmodule.c#L12602-L12605

In practice, this can happen if trailers is not valid for example. The function (legitimately) raises a TypeError, but:

  • There's a leak: iov_setup allocated an iovec array and a Py_buffer array with PyMem_New. (A call to iov_cleanup would free them, but it is not called.)
  • The input is left in an inconsistent state: PyObject_GetBuffer exported each header buffer. Without PyBuffer_Release, that export stays for the lifetime of the process, so for example a bytearray in headers can no longer be resized after the failure.

The same happens on macOS when the header-size overflow check returns after iov_setup, and when iov_setup for trailers fails after headers were already exported.

Reproducer

import os
import tempfile

header = bytearray(b"header")
with tempfile.TemporaryFile() as src, tempfile.TemporaryFile() as dst:
    try:
        os.sendfile(
            dst.fileno(), src.fileno(), 0, 0,
            headers=[header], trailers=object(),
        )
    except TypeError as exc:
        print(exc)  # sendfile() trailers must be a sequence

header.append(0)
# BufferError: Existing exports of data: object cannot be re-sized

The leak is properly reported by ASan/LSan:

=================================================================
==81993==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 80 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100fbbc00 in iov_setup posixmodule.c:12077
    #2 0x000100fae4d0 in os_sendfile_impl posixmodule.c:12454
    #3 0x000100fae4d0 in os_sendfile posixmodule.c.h:8233
    #4 0x000100a6e23c in _PyObject_VectorcallTstate pycore_call.h:144
    #5 0x000100a6e23c in PyObject_Vectorcall call.c:327
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100ddb36c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796
    #17 0x000100f7b978 in pymain_main main.c:826

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100fbbbc0 in iov_setup posixmodule.c:12071
    #2 0x000100fae4d0 in os_sendfile_impl posixmodule.c:12454
    #3 0x000100fae4d0 in os_sendfile posixmodule.c.h:8233
    #4 0x000100a6e23c in _PyObject_VectorcallTstate pycore_call.h:144
    #5 0x000100a6e23c in PyObject_Vectorcall call.c:327
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100ddb36c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796
    #17 0x000100f7b978 in pymain_main main.c:826

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100bd3684 in _PyObject_MallocWithType pycore_object_alloc.h:46
    #2 0x000100bd3684 in _PyType_AllocNoTrack typeobject.c:2523
    #3 0x000100bd345c in PyType_GenericAlloc typeobject.c:2554
    #4 0x000100bdd884 in type_call typeobject.c:2467
    #5 0x000100a6cd90 in _PyObject_MakeTpCall call.c:242
    #6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #7 0x000100de4dc0 in _PyEval_EvalFrameDefault generated_cases.c.h:1846
    #8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #11 0x000100f19ea4 in run_mod pythonrun.c:1472
    #12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
    #14 0x000100f7aad0 in pymain_run_command main.c:262
    #15 0x000100f7aad0 in pymain_run_python main.c:706
    #16 0x000100f7aad0 in Py_RunMain main.c:796

Indirect leak of 39 byte(s) in 1 object(s) allocated from:
    #0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
    #1 0x000100a5c9a4 in _PyBytes_FromSize bytesobject.c:121
    #2 0x000100a5c9a4 in _PyBytes_Resize bytesobject.c:3356
    #3 0x000100a3620c in bytearray_resize_lock_held bytearrayobject.c:280
    #4 0x000100a37b50 in PyByteArray_Resize bytearrayobject.c:299
    #5 0x000100a37b50 in bytearray___init___impl bytearrayobject.c:1021
    #6 0x000100a37b50 in bytearray___init__ bytearrayobject.c.h:102
    #7 0x000100bddab4 in type_call typeobject.c:2479
    #8 0x000100a6cd90 in _PyObject_MakeTpCall call.c:242
    #9 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
    #10 0x000100de4dc0 in _PyEval_EvalFrameDefault generated_cases.c.h:1846
    #11 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
    #12 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
    #13 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
    #14 0x000100f19ea4 in run_mod pythonrun.c:1472
    #15 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
    #16 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573

SUMMARY: AddressSanitizer: 199 byte(s) leaked in 4 allocation(s).
CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs
  • gh-156288

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

从 Modules/posixmodule.c 中的 iov_setup 和 os_sendfile 开始,跟踪报告中确定的验证路径。使用提供的 macOS 复现程序,验证验证失败会释放 header 和 trailer 资源,使缓冲区保持可调整大小,并且不会泄漏分配;关联的 PR gh-156288 表明相关工作已经在进行中。

由索引模型根据 Issue 内容生成。

评估

技术栈
c, python
领域
operating-systems
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
描述清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。