python / python/cpython

PyFaulthandler: heap-buffer-overflow

Open
#124,001 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

extension-modules type-crash
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR metrics pending

Description

Bug report

Bug description:

Enviroment:

FROM ubuntu:24.04@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30
RUN apt-get update -y && \
    apt-get install -y \
        autoconf \
        build-essential \
        clang \
        clang-tools \
        curl \
        gcc \
        gdb \
        gnupg \
        libbz2-dev \
        liblzma-dev \
        libncurses5-dev \
        libncursesw5-dev \
        libreadline-dev \
        libsqlite3-dev \
        libssl-dev \
        lld \
        lsb-release \
        make \
        pkg-config \
        rsync \
        software-properties-common \
        tk-dev \
        wget \
        wget \
        xz-utils \
        zlib1g-dev

WORKDIR /src/

RUN wget https://apt.llvm.org/llvm.sh && \
    chmod +x llvm.sh && \
    /src/llvm.sh 19
RUN wget https://www.python.org/ftp/python/3.12.4/Python-3.12.4.tgz && \
    tar xzf Python-3.12.4.tgz && \
    rm Python-3.12.4.tgz
RUN wget https://www.python.org/ftp/python/3.13.0/Python-3.13.0b4.tgz && \
    tar xzf Python-3.13.0b4.tgz && \
    rm Python-3.13.0b4.tgz
COPY build_scripts .

Build script:

#!/bin/bash
set -x
set -e
# print OS version
cat /etc/os-release
# print current env
env
export CC=/usr/bin/gcc
export CXX=/usr/bin/g++
export CFLAGS="-g -fsanitize=hwaddress"
export CCFLAGS="-g -fsanitize=hwaddress"
export CXXFLAGS="-g -fsanitize=hwaddress"
export CPPFLAGS="-g -fsanitize=hwaddress"
export LDFLAGS="-fsanitize=hwaddress"
export HWASAN_OPTIONS="detect_leaks=0:allocator_may_return_null=1:handle_segv=0"
cd /src/Python-3.13.0b4
set +e
if ! ./configure --with-assertions --with-pydebug; then
    cat config.log
    exit 1
else
    make
    ./python -m test -uall # <----
fi

Log (full):

------------ CUT ------------
test.test_concurrent_futures.test_shutdown

test test.test_concurrent_futures.test_shutdown failed -- Traceback (most recent call last):
  File "/src/Python-3.13.0b4/Lib/test/test_concurrent_futures/test_shutdown.py", line 49, in test_interpreter_shutdown
    self.assertFalse(err)
    ~~~~~~~~~~~~~~~~^^^^^
AssertionError: b'==10549==ERROR: HWAddressSanitizer: tag-mismatch on address 0xeeadffe20020 at pc 0xffff86e1e080
WRITE of size 25208 at 0xeeadffe20020 tags: d6/00 (ptr/mem) in thread T0
Invalid access starting at offset 20352
    #0 0xffff86e1e080 in SigTrap<(__hwasan::ErrorAction)0, (__hwasan::AccessType)1> ../../../../src/libsanitizer/hwasan/hwasan_checks.h:107
    #1 0xffff86e1e080 in CheckAddressSized<(__hwasan::ErrorAction)0, (__hwasan::AccessType)1> ../../../../src/libsanitizer/hwasan/hwasan_checks.h:187
    #2 0xffff86e1e080 in __hwasan_storeN ../../../../src/libsanitizer/hwasan/hwasan.cpp:586
    #3 0xaaaac2c25d04 in memset /usr/include/aarch64-linux-gnu/bits/string_fortified.h:59
    #4 0xaaaac2c25d04 in fill_mem_debug Objects/obmalloc.c:2632
    #5 0xaaaac2c27c40 in _PyMem_DebugRawFree Objects/obmalloc.c:2761
    #6 0xaaaac2c27d60 in _PyMem_DebugFree Objects/obmalloc.c:2899
    #7 0xaaaac2c46898 in PyMem_Free Objects/obmalloc.c:1018
    #8 0xaaaac2e53fe4 in _PyFaulthandler_Fini Modules/faulthandler.c:1409
    #9 0xaaaac2e0a9e0 in _Py_Finalize Python/pylifecycle.c:2123
    #10 0xaaaac2e0b1a4 in Py_Exit Python/pylifecycle.c:3392
    #11 0xaaaac2e13ed0 in handle_system_exit Python/pythonrun.c:604
    #12 0xaaaac2e14368 in _PyErr_PrintEx Python/pythonrun.c:613
    #13 0xaaaac2e1473c in PyErr_PrintEx Python/pythonrun.c:690
    #14 0xaaaac2e14754 in PyErr_Print Python/pythonrun.c:696
    #15 0xaaaac2e15148 in _PyRun_SimpleStringFlagsWithName Python/pythonrun.c:521
    #16 0xaaaac2e4d63c in pymain_run_command Modules/main.c:253
    #17 0xaaaac2e4e97c in pymain_run_python Modules/main.c:683
    #18 0xaaaac2e4ef80 in Py_RunMain Modules/main.c:771
    #19 0xaaaac2e4f0bc in pymain_main Modules/main.c:801
    #20 0xaaaac2e4f2f8 in Py_BytesMain Modules/main.c:825
    #21 0xaaaac2ac3c5c in main Programs/python.c:15
    #22 0xffff86bb84c0 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #23 0xffff86bb8594 in __libc_start_main_impl ../csu/libc-start.c:360
    #24 0xaaaac2ac3b6c in _start (/src/Python-3.13.0b4/python+0x83b6c) (BuildId: 7f970e5620c31cf1b742716ca1bfcc478b4f4710)

[0xeeadffe20000,0xeeadffe27000) is a small allocated heap chunk; size: 28672 offset: 32

Cause: heap-buffer-overflow
0xeeadffe20020 is located 32 bytes inside a 25248-byte region [0xeeadffe20000,0xeeadffe262a0)
allocated by thread T0 here:
    #0 0xffff86e16768 in __sanitizer_malloc ../../../../src/libsanitizer/hwasan/hwasan_allocation_functions.cpp:151
    #1 0xaaaac2c26dc4 in _PyMem_RawMalloc Objects/obmalloc.c:62
    #2 0xaaaac2c25e30 in _PyMem_DebugRawAlloc Objects/obmalloc.c:2694
    #3 0xaaaac2c25e80 in _PyMem_DebugRawMalloc Objects/obmalloc.c:2727
    #4 0xaaaac2c45aa8 in PyMem_RawMalloc Objects/obmalloc.c:948
    #5 0xaaaac2c48418 in _PyObject_Malloc Objects/obmalloc.c:2140
    #6 0xaaaac2c25e30 in _PyMem_DebugRawAlloc Objects/obmalloc.c:2694
    #7 0xaaaac2c25e80 in _PyMem_DebugRawMalloc Objects/obmalloc.c:2727
    #8 0xaaaac2c27cd8 in _PyMem_DebugMalloc Objects/obmalloc.c:2884
    #9 0xaaaac2c467e8 in PyMem_Malloc Objects/obmalloc.c:989
    #10 0xaaaac2e52468 in faulthandler_allocate_stack Modules/faulthandler.c:417
    #11 0xaaaac2e527c0 in faulthandler_enable Modules/faulthandler.c:447
    #12 0xaaaac2e530f4 in faulthandler_py_enable Modules/faulthandler.c:519
    #13 0xaaaac2c19ee8 in cfunction_call Objects/methodobject.c:540
    #14 0xaaaac2b9defc in _PyObject_MakeTpCall Objects/call.c:242
    #15 0xaaaac2b9e178 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:166
    #16 0xaaaac2b9e1a8 in PyObject_CallNoArgs Objects/call.c:106
    #17 0xaaaac2e53d0c in faulthandler_init_enable Modules/faulthandler.c:1322
    #18 0xaaaac2e53e3c in _PyFaulthandler_Init Modules/faulthandler.c:1357
    #19 0xaaaac2e07fe0 in init_interp_main Python/pylifecycle.c:1194

Thread: T0 0xeffe00002000 stack: [0xffffe2ff0000,0xffffe37f0000) sz: 8388608 tls: [0xffff874ca660,0xffff874cb520)
Memory tags around the buggy address (one tag corresponds to 16 bytes):
  0xeeadffe24700: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe24800: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe24900: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe24a00: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe24b00: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe24c00: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe24d00: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe24e00: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
=>0xeeadffe24f00: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 [00] 00  d6  d6  d6  d6 
  0xeeadffe25000: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe25100: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe25200: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe25300: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe25400: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe25500: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe25600: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
  0xeeadffe25700: d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6  d6 
Tags for short granules around the buggy address (one tag corresponds to 16 bytes):
  0xeeadffe24e00: ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  .. 
=>0xeeadffe24f00: ..  ..  ..  ..  ..  ..  ..  ..  ..  .. [..] ..  ..  ..  ..  .. 
  0xeeadffe25000: ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  .. 
See https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html#short-granules for a description of short granule tags
SUMMARY: HWAddressSanitizer: tag-mismatch ../../../../src/libsanitizer/hwasan/hwasan_checks.h:107 in SigTrap<(__hwasan::ErrorAction)0, (__hwasan::AccessType)1>
' is not false
------------ CUT ------------

------------ CUT ------------
test.test_multiprocessing_forkserver.test_misc
ok

==17050==ERROR: HWAddressSanitizer: tag-mismatch on address 0xeeadffe20020 at pc 0xffffa2c1e080
WRITE of size 25208 at 0xeeadffe20020 tags: c0/00 (ptr/mem) in thread T0
Invalid access starting at offset 20352
    #0 0xffffa2c1e080 in SigTrap<(__hwasan::ErrorAction)0, (__hwasan::AccessType)1> ../../../../src/libsanitizer/hwasan/hwasan_checks.h:107
    #1 0xffffa2c1e080 in CheckAddressSized<(__hwasan::ErrorAction)0, (__hwasan::AccessType)1> ../../../../src/libsanitizer/hwasan/hwasan_checks.h:187
    #2 0xffffa2c1e080 in __hwasan_storeN ../../../../src/libsanitizer/hwasan/hwasan.cpp:586
    #3 0xaaaad2425d04 in memset /usr/include/aarch64-linux-gnu/bits/string_fortified.h:59
    #4 0xaaaad2425d04 in fill_mem_debug Objects/obmalloc.c:2632
    #5 0xaaaad2427c40 in _PyMem_DebugRawFree Objects/obmalloc.c:2761
    #6 0xaaaad2427d60 in _PyMem_DebugFree Objects/obmalloc.c:2899
    #7 0xaaaad2446898 in PyMem_Free Objects/obmalloc.c:1018
    #8 0xaaaad2653fe4 in _PyFaulthandler_Fini Modules/faulthandler.c:1409
    #9 0xaaaad260a9e0 in _Py_Finalize Python/pylifecycle.c:2123
    #10 0xaaaad260b1a4 in Py_Exit Python/pylifecycle.c:3392
    #11 0xaaaad2613ed0 in handle_system_exit Python/pythonrun.c:604
    #12 0xaaaad2614368 in _PyErr_PrintEx Python/pythonrun.c:613
    #13 0xaaaad261473c in PyErr_PrintEx Python/pythonrun.c:690
    #14 0xaaaad2614754 in PyErr_Print Python/pythonrun.c:696
    #15 0xaaaad2615148 in _PyRun_SimpleStringFlagsWithName Python/pythonrun.c:521
    #16 0xaaaad264d63c in pymain_run_command Modules/main.c:253
    #17 0xaaaad264e97c in pymain_run_python Modules/main.c:683
    #18 0xaaaad264ef80 in Py_RunMain Modules/main.c:771
    #19 0xaaaad264f0bc in pymain_main Modules/main.c:801
    #20 0xaaaad264f2f8 in Py_BytesMain Modules/main.c:825
    #21 0xaaaad22c3c5c in main Programs/python.c:15
    #22 0xffffa29b84c0 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #23 0xffffa29b8594 in __libc_start_main_impl ../csu/libc-start.c:360
    #24 0xaaaad22c3b6c in _start (/src/Python-3.13.0b4/python+0x83b6c) (BuildId: 7f970e5620c31cf1b742716ca1bfcc478b4f4710)

[0xeeadffe20000,0xeeadffe27000) is a small allocated heap chunk; size: 28672 offset: 32

Cause: heap-buffer-overflow
0xeeadffe20020 is located 32 bytes inside a 25248-byte region [0xeeadffe20000,0xeeadffe262a0)
allocated by thread T0 here:
    #0 0xffffa2c16768 in __sanitizer_malloc ../../../../src/libsanitizer/hwasan/hwasan_allocation_functions.cpp:151
    #1 0xaaaad2426dc4 in _PyMem_RawMalloc Objects/obmalloc.c:62
    #2 0xaaaad2425e30 in _PyMem_DebugRawAlloc Objects/obmalloc.c:2694
    #3 0xaaaad2425e80 in _PyMem_DebugRawMalloc Objects/obmalloc.c:2727
    #4 0xaaaad2445aa8 in PyMem_RawMalloc Objects/obmalloc.c:948
    #5 0xaaaad2448418 in _PyObject_Malloc Objects/obmalloc.c:2140
    #6 0xaaaad2425e30 in _PyMem_DebugRawAlloc Objects/obmalloc.c:2694
    #7 0xaaaad2425e80 in _PyMem_DebugRawMalloc Objects/obmalloc.c:2727
    #8 0xaaaad2427cd8 in _PyMem_DebugMalloc Objects/obmalloc.c:2884
    #9 0xaaaad24467e8 in PyMem_Malloc Objects/obmalloc.c:989
    #10 0xaaaad2652468 in faulthandler_allocate_stack Modules/faulthandler.c:417
    #11 0xaaaad26527c0 in faulthandler_enable Modules/faulthandler.c:447
    #12 0xaaaad26530f4 in faulthandler_py_enable Modules/faulthandler.c:519
    #13 0xaaaad2419ee8 in cfunction_call Objects/methodobject.c:540
    #14 0xaaaad239defc in _PyObject_MakeTpCall Objects/call.c:242
    #15 0xaaaad239e178 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:166
    #16 0xaaaad239e1a8 in PyObject_CallNoArgs Objects/call.c:106
    #17 0xaaaad2653d0c in faulthandler_init_enable Modules/faulthandler.c:1322
    #18 0xaaaad2653e3c in _PyFaulthandler_Init Modules/faulthandler.c:1357
    #19 0xaaaad2607fe0 in init_interp_main Python/pylifecycle.c:1194

Thread: T0 0xeffe00002000 stack: [0xffffe1797000,0xffffe1f97000) sz: 8388608 tls: [0xffffa32fa660,0xffffa32fb520)
Memory tags around the buggy address (one tag corresponds to 16 bytes):
  0xeeadffe24700: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe24800: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe24900: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe24a00: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe24b00: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe24c00: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe24d00: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe24e00: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
=>0xeeadffe24f00: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 [00] 00  c0  c0  c0  c0 
  0xeeadffe25000: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe25100: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe25200: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe25300: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe25400: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe25500: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe25600: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
  0xeeadffe25700: c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0  c0 
Tags for short granules around the buggy address (one tag corresponds to 16 bytes):
  0xeeadffe24e00: ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  .. 
=>0xeeadffe24f00: ..  ..  ..  ..  ..  ..  ..  ..  ..  .. [..] ..  ..  ..  ..  .. 
  0xeeadffe25000: ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  .. 
See https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html#short-granules for a description of short granule tags
SUMMARY: HWAddressSanitizer: tag-mismatch ../../../../src/libsanitizer/hwasan/hwasan_checks.h:107 in SigTrap<(__hwasan::ErrorAction)0, (__hwasan::AccessType)1>

test test.test_multiprocessing_forkserver.test_misc failed -- Traceback (most recent call last):
  File "/src/Python-3.13.0b4/Lib/test/_test_multiprocessing.py", line 5576, in test_preload_resources
    self.fail("failed spawning forkserver or grandchild")
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: failed spawning forkserver or grandchild
------------ CUT ------------

Also reproduced (full table):

OS VERSION LOGS
fedora Python 3.12.4 build_004.sh logs
fedora Python 3.13.0b4 build_041.sh logs
ubuntu Python 3.12.4 build_088.sh logs
ubuntu Python 3.13.0b4 build_149.sh logs (this issue)
CPython versions tested on:

3.13

Operating systems tested on:

Linux

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Modules/faulthandler.c, especially faulthandler_allocate_stack(), faulthandler_enable(), and _PyFaulthandler_Fini() named in the trace, alongside Objects/obmalloc.c. Reproduce with the provided Ubuntu 24.04/AArch64 HWASAN build and ./python -m test -uall; done means test.test_concurrent_futures.test_shutdown no longer reports the heap-buffer-overflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.