`os.sendfile` leaks header buffer exports on validation errors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
Bug report
Bug description:
On macOS (and as far as I know FreeBSD), os.sendfile takes (among others) two headers and trailers parameters. Calling iov_setup acquires Py_buffer exports for headers, but several validation steps that come after can error and return before iov_cleanup is ever called.
The setup is done here:
The early returns after that never call iov_cleanup:
iov_cleanup is only called for trailers and headers after the syscall:
In practice, this can happen if trailers is not valid for example. The function (legitimately) raises a TypeError, but:
- There's a leak:
iov_setupallocated aniovecarray and aPy_bufferarray withPyMem_New. (A call toiov_cleanupwould free them, but it is not called.) - The input is left in an inconsistent state:
PyObject_GetBufferexported each header buffer. WithoutPyBuffer_Release, that export stays for the lifetime of the process, so for example abytearrayinheaderscan no longer be resized after the failure.
The same happens on macOS when the header-size overflow check returns after iov_setup, and when iov_setup for trailers fails after headers were already exported.
Reproducer
import os
import tempfile
header = bytearray(b"header")
with tempfile.TemporaryFile() as src, tempfile.TemporaryFile() as dst:
try:
os.sendfile(
dst.fileno(), src.fileno(), 0, 0,
headers=[header], trailers=object(),
)
except TypeError as exc:
print(exc) # sendfile() trailers must be a sequence
header.append(0)
# BufferError: Existing exports of data: object cannot be re-sized
The leak is properly reported by ASan/LSan:
=================================================================
==81993==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 80 byte(s) in 1 object(s) allocated from:
#0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
#1 0x000100fbbc00 in iov_setup posixmodule.c:12077
#2 0x000100fae4d0 in os_sendfile_impl posixmodule.c:12454
#3 0x000100fae4d0 in os_sendfile posixmodule.c.h:8233
#4 0x000100a6e23c in _PyObject_VectorcallTstate pycore_call.h:144
#5 0x000100a6e23c in PyObject_Vectorcall call.c:327
#6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
#7 0x000100ddb36c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
#8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
#9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
#10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
#11 0x000100f19ea4 in run_mod pythonrun.c:1472
#12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
#13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
#14 0x000100f7aad0 in pymain_run_command main.c:262
#15 0x000100f7aad0 in pymain_run_python main.c:706
#16 0x000100f7aad0 in Py_RunMain main.c:796
#17 0x000100f7b978 in pymain_main main.c:826
Direct leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
#1 0x000100fbbbc0 in iov_setup posixmodule.c:12071
#2 0x000100fae4d0 in os_sendfile_impl posixmodule.c:12454
#3 0x000100fae4d0 in os_sendfile posixmodule.c.h:8233
#4 0x000100a6e23c in _PyObject_VectorcallTstate pycore_call.h:144
#5 0x000100a6e23c in PyObject_Vectorcall call.c:327
#6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
#7 0x000100ddb36c in _PyEval_EvalFrameDefault generated_cases.c.h:3325
#8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
#9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
#10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
#11 0x000100f19ea4 in run_mod pythonrun.c:1472
#12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
#13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
#14 0x000100f7aad0 in pymain_run_command main.c:262
#15 0x000100f7aad0 in pymain_run_python main.c:706
#16 0x000100f7aad0 in Py_RunMain main.c:796
#17 0x000100f7b978 in pymain_main main.c:826
Indirect leak of 64 byte(s) in 1 object(s) allocated from:
#0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
#1 0x000100bd3684 in _PyObject_MallocWithType pycore_object_alloc.h:46
#2 0x000100bd3684 in _PyType_AllocNoTrack typeobject.c:2523
#3 0x000100bd345c in PyType_GenericAlloc typeobject.c:2554
#4 0x000100bdd884 in type_call typeobject.c:2467
#5 0x000100a6cd90 in _PyObject_MakeTpCall call.c:242
#6 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
#7 0x000100de4dc0 in _PyEval_EvalFrameDefault generated_cases.c.h:1846
#8 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
#9 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
#10 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
#11 0x000100f19ea4 in run_mod pythonrun.c:1472
#12 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
#13 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
#14 0x000100f7aad0 in pymain_run_command main.c:262
#15 0x000100f7aad0 in pymain_run_python main.c:706
#16 0x000100f7aad0 in Py_RunMain main.c:796
Indirect leak of 39 byte(s) in 1 object(s) allocated from:
#0 0x0001020e4e24 in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x54e24)
#1 0x000100a5c9a4 in _PyBytes_FromSize bytesobject.c:121
#2 0x000100a5c9a4 in _PyBytes_Resize bytesobject.c:3356
#3 0x000100a3620c in bytearray_resize_lock_held bytearrayobject.c:280
#4 0x000100a37b50 in PyByteArray_Resize bytearrayobject.c:299
#5 0x000100a37b50 in bytearray___init___impl bytearrayobject.c:1021
#6 0x000100a37b50 in bytearray___init__ bytearrayobject.c.h:102
#7 0x000100bddab4 in type_call typeobject.c:2479
#8 0x000100a6cd90 in _PyObject_MakeTpCall call.c:242
#9 0x000100dc4e0c in _Py_VectorCallInstrumentation_StackRefSteal ceval.c:775
#10 0x000100de4dc0 in _PyEval_EvalFrameDefault generated_cases.c.h:1846
#11 0x000100dc3ae0 in _PyEval_EvalFrame pycore_ceval.h:122
#12 0x000100dc3ae0 in _PyEval_Vector ceval.c:2156
#13 0x000100dc3ae0 in PyEval_EvalCode ceval.c:686
#14 0x000100f19ea4 in run_mod pythonrun.c:1472
#15 0x000100f1604c in _PyRun_StringFlagsWithName pythonrun.c:1260
#16 0x000100f15bd4 in _PyRun_SimpleStringFlagsWithName pythonrun.c:573
SUMMARY: AddressSanitizer: 199 byte(s) leaked in 4 allocation(s).
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Linked PRs
- gh-156288
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in Modules/posixmodule.c at iov_setup and os_sendfile, following the validation paths identified in the report. Use the provided macOS reproducer to verify that validation failures release header and trailer resources, leave buffers resizable, and do not leak allocations; linked PR gh-156288 indicates work is already underway.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, python
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100