numpy / numpy/numpy

segfault when importing numpy after reinitializing Python

Open
#11,925 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Embedded
Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

I embedded Python into a C program. At certain points in time, the C program resets the interpreter by calling Py_Finalize + Py_Initialize. When the evaluated code imports NumPy before and after the reset, then the second import fails with a segmentation fault.

Reproducing code example:

At first the example C program initializes Python, imports NumPy, and then finalizes Python.
Then it reinitializes Python and imports NumPy again. When importing NumPy
for the second time, the import fails with a segmentation fault.

Tested with Debian 9, Python 3.7.0 valgrind friendly debug build, NumPy debug build of latest
version from github (1.16.0.dev0+e796449). The bug reproduces on Mac (tested with Python 3.5.1,
NumPy 1.11.0) and Windows (Tested with Python 3.7.0 from Python.org, NumPy 1.14.5).

/******************************************************************************
    Test: Initialize NumPy before and after Python reinitialization
******************************************************************************/

#include "Python.h"

int main(int arc, char *argv[]) {
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) {
        fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
        exit(1);
    }

    Py_SetProgramName(program);

    // Initialize Python, import NumPy, finalize Python. First time.
    Py_Initialize();
    PyRun_SimpleString("import sys; print('Python', sys.version)");
    PyRun_SimpleString("import numpy; print('Numpy', numpy.__version__)");
    Py_Finalize();

    // Initialize Python, import NumPy, finalize Python. Second time.
    Py_Initialize();
    PyRun_SimpleString("import numpy;"); // <-- SEGMENTATION FAULT.
    Py_Finalize();

    PyMem_RawFree(program);
    return 0;
}
Error message:

Segmentation fault on evaluation of "import numpy" after reinitialization of Python. Output of valgrind:

==4346== Memcheck, a memory error detector
==4346== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==4346== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==4346== Command: ./main
==4346==
Python 3.7.0 (default, Aug 27 2018, 19:39:47)
[GCC 6.3.0 20170516]
Numpy 1.16.0.dev0+e796449
==4346== Invalid read of size 8
==4346==    at 0x6CEFDC4: PyArray_Item_INCREF (refcount.c:35)
==4346==    by 0x6CF37A8: PyArray_FromScalar (scalarapi.c:335)
==4346==    by 0x6CF5674: gentype_nonzero_number (scalartypes.c.src:349)
==4346==    by 0x4F12AA8: PyObject_IsTrue (object.c:1384)
==4346==    by 0x4FC1995: _PyEval_EvalFrameDefault (ceval.c:2654)
==4346==    by 0x4FB7EE1: PyEval_EvalFrameEx (ceval.c:547)
==4346==    by 0x4EC7C0C: function_code_fastcall (call.c:283)
==4346==    by 0x4EC87CF: _PyFunction_FastCallKeywords (call.c:408)
==4346==    by 0x4FC42FD: call_function (ceval.c:4586)
==4346==    by 0x4FC42FD: _PyEval_EvalFrameDefault (ceval.c:3117)
==4346==    by 0x4FB7EE1: PyEval_EvalFrameEx (ceval.c:547)
==4346==    by 0x4EC7C0C: function_code_fastcall (call.c:283)
==4346==    by 0x4EC846B: _PyFunction_FastCallDict (call.c:322)
==4346==  Address 0x1a is not stack'd, malloc'd or (recently) free'd
==4346==
==4346==
==4346== Process terminating with default action of signal 11 (SIGSEGV)
==4346==  Access not within mapped region at address 0x1A
==4346==    at 0x6CEFDC4: PyArray_Item_INCREF (refcount.c:35)
==4346==    by 0x6CF37A8: PyArray_FromScalar (scalarapi.c:335)
==4346==    by 0x6CF5674: gentype_nonzero_number (scalartypes.c.src:349)
==4346==    by 0x4F12AA8: PyObject_IsTrue (object.c:1384)
==4346==    by 0x4FC1995: _PyEval_EvalFrameDefault (ceval.c:2654)
==4346==    by 0x4FB7EE1: PyEval_EvalFrameEx (ceval.c:547)
==4346==    by 0x4EC7C0C: function_code_fastcall (call.c:283)
==4346==    by 0x4EC87CF: _PyFunction_FastCallKeywords (call.c:408)
==4346==    by 0x4FC42FD: call_function (ceval.c:4586)
==4346==    by 0x4FC42FD: _PyEval_EvalFrameDefault (ceval.c:3117)
==4346==    by 0x4FB7EE1: PyEval_EvalFrameEx (ceval.c:547)
==4346==    by 0x4EC7C0C: function_code_fastcall (call.c:283)
==4346==    by 0x4EC846B: _PyFunction_FastCallDict (call.c:322)
==4346==  If you believe this happened as a result of a stack
==4346==  overflow in your program's main thread (unlikely but
==4346==  possible), you can try to increase the size of the
==4346==  main thread stack using the --main-stacksize= flag.
==4346==  The main thread stack size used in this run was 8388608.
==4346==
==4346== HEAP SUMMARY:
==4346==     in use at exit: 6,190,252 bytes in 34,347 blocks
==4346==   total heap usage: 215,244 allocs, 180,897 frees, 45,494,245 bytes allocated
==4346==
==4346== LEAK SUMMARY:
==4346==    definitely lost: 144 bytes in 2 blocks
==4346==    indirectly lost: 0 bytes in 0 blocks
==4346==      possibly lost: 6,055,624 bytes in 31,510 blocks
==4346==    still reachable: 134,484 bytes in 2,835 blocks
==4346==         suppressed: 0 bytes in 0 blocks
==4346== Rerun with --leak-check=full to see details of leaked memory
==4346==
==4346== For counts of detected and suppressed errors, rerun with: -v
==4346== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Killed
Numpy/Python version information:

Python 3.7.0 (default, Aug 27 2018, 19:39:47) [GCC 6.3.0 20170516]
Numpy 1.16.0.dev0+e796449

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the embedded C reproducer in the issue and run it under Valgrind to confirm the second NumPy import failure. Trace the reported stack frames in refcount.c, scalarapi.c, and scalartypes.c.src; done means reinitializing Python and importing NumPy again no longer segfaults on the reported platforms.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.