indygreg / indygreg/PyOxidizer

Segmentation fault if built extension is located in sys_paths

Open
#212 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

If one of the entries in sys_paths contain a built extension, and it is imported, the binary always seg faults. e.g. scandir in repl mode

```gdb
#0 0x00007ffff700cab8 in PyModule_Create2 () from /usr/lib64/libpython3.7m.so.1.0
#1 0x00007ffff7f82503 in PyInit__scandir () from /usr/lib64/python3.7/site-packages/_scandir.cpython-37m-x86_64-linux-gnu.so
#2 0x0000555555803896 in _PyImport_LoadDynamicModuleWithSpec (
spec=, origin='/usr/lib64/python3.7/site-packages/_scandir.cpython-37m-x86_64-linux-gnu.so', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached=None) at remote 0x7ffff72c2190>, fp=) at ./Python/importdl.c:159
#3 0x0000555555803327 in _imp_create_dynamic_impl (module=,
spec=, origin='/usr/lib64/python3.7/site-packages/_scandir.cpython-37m-x86_64-linux-gnu.so', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached=None) at remote 0x7ffff72c2190>, file=) at Python/import.c:2170
#4 _imp_create_dynamic (module=, args=, nargs=) at Python/clinic/import.c.h:289
#5 0x00005555557406eb in _PyMethodDef_RawFastCallDict (method=0x55555a2a0d80 , self=, args=0x7ffff72b74e8, nargs=1, kwargs={})
at Objects/call.c:530
#6 0x00005555557401b4 in _PyCFunction_FastCallDict (func=, args=0x7ffff72b74e8, nargs=33, kwargs={})
at Objects/call.c:586
#7 PyCFunction_Call (func=, args=, kwargs={}) at Objects/call.c:791
#8 0x00005555557d73f2 in do_call_core (func=,
callargs=(, origin='/usr/lib64/python3.7/site-packages/_scandir.cpython-37m-x86_64-linux-gnu.so', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached=None) at remote 0x7ffff72c2190>,), kwdict=) at Python/ceval.c:4641
#9 _PyEval_EvalFrameDefault (
f=Frame 0x7ffff736ae50, for file , line 219, in _call_with_frames_removed (f=, args=(, origin='/usr/lib64/python3.7/site-packages/_scandir.cpython-37m-x86_64-linux-gnu.so', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached=None) at remote 0x7ffff72c2190>,), kwds={}), throwflag=) at Python/ceval.c:3191
#10 0x00005555557da8dd in PyEval_EvalFrameEx (f=, throwflag=0) at Python/ceval.c:547
#11 _PyEval_EvalCodeWithName (_co=, globals=, locals=, args=, argcount=2, kwnames=0x0, kwargs=0x7ffff792bac0, kwcount=0, kwstep=1,
defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name='_call_with_frames_removed', qualname='_call_with_frames_removed') at Python/ceval.c:3930
#12 0x000055555573feb8 in _PyFunction_FastCallKeywords (func=, stack=0x21, nargs=0, kwnames=) at Objects/call.c:433
#13 0x00005555557d9b62 in call_function (pp_stack=0x7fffffff9a70, oparg=, kwnames=0x0) at Python/ceval.c:4616
#14 0x00005555557d7002 in _PyEval_EvalFrameDefault (
f=Frame 0x7ffff792b920, for file , line 1043, in create_module (self=, spec=, origin='/usr/lib64/python3.7/site-packages/_scandir.cpython-37m-x86_64-linux-gnu.so', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached=None) at remote 0x7ffff72c2190>), throwflag=) at Python/ceval.c:3093
#15 0x00005555557403df in function_code_fastcall (co=, args=, nargs=, globals=) at Objects/call.c:283
#16 0x00005555557d9b62 in call_function (pp_stack=0x7fffffff9c60, oparg=, kwnames=0x0) at Python/ceval.c:4616
#17 0x00005555557d6fec in _PyEval_EvalFrameDefault (
f=Frame 0x7ffff739d750, for file , line 583, in module_from_spec (spec=, origin='/usr/lib64/python3.7/site-packages/_scandir.cpython-37m-x86_64-linux-gnu.so', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached=None) at remote 0x7ffff72c2190>, module=None), throwflag=) at Python/ceval.c:3110
#18 0x00005555557403df in function_code_fastcall (co=, args=, nargs=, globals=) at Objects/call.c:283
#19 0x00005555557d9b62 in call_function (pp_stack=0x7fffffff9eb0, oparg=, kwnames=0x0) at Python/ceval.c:4616
#20 0x00005555557d708e in _PyEval_EvalFrameDefault (
f=Frame 0x7ffff72b8050, for file , line 670, in _load_unlocked (spec=, origin='/usr/lib64/python3.7/site-packages/_scandir.cpython-37m-x86_64-linux-gnu.so', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached=None) at remote 0x7ffff72c2190>), throwflag=) at Python/ceval.c:3124
#21 0x00005555557403df in function_code_fastcall (co=, args=, nargs=, globals=) at Objects/call.c:283
#22 0x00005555557d9b62 in call_function (pp_stack=0x7fffffffa0a0, oparg=, kwnames=0x0) at Python/ceval.c:4616
#23 0x00005555557d708e in _PyEval_EvalFrameDefault (
f=Frame 0x7ffff792e8a0, for file , line 967, in _find_and_load_unlocked (name='_scandir', import_= for more, q to quit, c to continue without paging--
f7fbde30>, path=None, parent='', spec=, origin='/usr/lib64/python3.7/site-packages/_scandir.cpython-37m-x86_64-linux-gnu.so', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached=None) at remote 0x7ffff72c2190>), throwflag=) at Python/ceval.c:3124
#24 0x00005555557403df in function_code_fastcall (co=, args=, nargs=, globals=) at Objects/call.c:283
#25 0x00005555557d9b62 in call_function (pp_stack=0x7fffffffa290, oparg=, kwnames=0x0) at Python/ceval.c:4616
#26 0x00005555557d708e in _PyEval_EvalFrameDefault (
f=Frame 0x7ffff728e650, for file , line 983, in _find_and_load (name='_scandir', import_=, module=), throwflag=) at Python/ceval.c:3124
#27 0x00005555557403df in function_code_fastcall (co=, args=, nargs=, globals=) at Objects/call.c:283
#28 0x0000555555741aae in object_vacall (callable=, vargs=0x7fffffffa3f0) at Objects/call.c:1202
#29 0x0000555555741b80 in _PyObject_CallMethodIdObjArgs (obj=, name=) at Objects/call.c:1252
#30 0x0000555555802301 in import_find_and_load (abs_name=) at Python/import.c:1648
#31 PyImport_ImportModuleLevelObject (name='_scandir', globals=, locals=, fromlist=None, level=0) at Python/import.c:1760
#32 0x00005555557d59eb in import_name (f=, name='_scandir', fromlist=None, level=0) at Python/ceval.c:4770
#33 _PyEval_EvalFrameDefault (f=Frame 0x7ffff78feda0, for file /usr/lib64/python3.7/site-packages/scandir.py, line 29, in (), throwflag=) at Python/ceval.c:2600
#34 0x00005555557da8dd in PyEval_EvalFrameEx (f=, throwflag=0) at Python/ceval.c:547
#35 _PyEval_EvalCodeWithName (_co=, globals=, locals=, args=, argcount=0, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0,
defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:3930
#36 0x00005555557d0cbf in PyEval_EvalCodeEx (_co=, globals=, locals=0x0, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0,
defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3959
#37 PyEval_EvalCode (co=, globals=, locals=0x0) at Python/ceval.c:524
#38 0x00005555557ce27d in builtin_exec_impl (module=, source=,
globals={'__name__': 'scandir', '__doc__': 'scandir, a better directory iterator and faster os.walk(), now in the Python 3.5 stdlib\n\nscandir() is a generator version of os.listdir() that returns an\niterator over files in a directory, and also exposes the extra\ninformation most OSes provide while iterating files in a directory\n(such as type and stat information).\n\nThis module also includes a version of os.walk() that uses scandir()\nto speed it up significantly.\n\nSee README.md or https://github.com/benhoyt/scandir for rationale and\ndocs, or read PEP 471 (https://www.python.org/dev/peps/pep-0471/) for\nmore details on its inclusion into Python 3.5\n\nscandir is released under the new BSD 3-clause license. See\nLICENSE.txt for the full license text.\n', '__package__': '', '__loader__': , '__spec__': , origin='/usr/lib64/python3.7/site-packages/scandir.py', loa...(truncated),
locals={'__name__': 'scandir', '__doc__': 'scandir, a better directory iterator and faster os.walk(), now in the Python 3.5 stdlib\n\nscandir() is a generator version of os.listdir() that returns an\niterator over files in a directory, and also exposes the extra\ninformation most OSes provide while iterating files in a directory\n(such as type and stat information).\n\nThis module also includes a version of os.walk() that uses scandir()\nto speed it up significantly.\n\nSee README.md or https://github.com/benhoyt/scandir for rationale and\ndocs, or read PEP 471 (https://www.python.org/dev/peps/pep-0471/) for\nmore details on its inclusion into Python 3.5\n\nscandir is released under the new BSD 3-clause license. See\nLICENSE.txt for the full license text.\n', '__package__': '', '__loader__': , '__spec__': , origin='/usr/lib64/python3.7/site-packages/scandir.py', loa...(truncated)) at Python/bltinmodule.c:1079
#39 builtin_exec (module=, args=, nargs=) at Python/clinic/bltinmodule.c.h:283
#40 0x00005555557406eb in _PyMethodDef_RawFastCallDict (method=0x55555a29d5d0 , self=, args=0x7ffff7366b08, nargs=2, kwargs={})
at Objects/call.c:530
#41 0x00005555557401b4 in _PyCFunction_FastCallDict (func=, args=0x7ffff7366b08, nargs=33, kwargs={}) at Objects/call.c:586
#42 PyCFunction_Call (func=, args=, kwargs={}) at Objects/call.c:791
#43 0x00005555557d73f2 in do_call_core (func=,
callargs=(, {'__name__': 'scandir', '__doc__': 'scandir, a better directory iterator and faster os.walk(), now in the Python 3.5 stdlib\n\nscandir() is a generator version of os.listdir() that returns an\niterator over files in a directory, and also exposes the extra\ninformation most OSes provide while iterating files in a directory\n(such as type and stat information).\n\nThis module also includes a version of os.walk() that uses scandir()\nto speed it up significantly.\n\nSee README.md or https://github.com/benhoyt/scandir for rationale and\ndocs, or read PEP 471 (https://www.python.org/dev/peps/pep-0471/) for\nmore details on its inclusion into Python 3.5\n\nscandir is released under the new BSD 3-clause license. See\nLICENSE.txt for the full license text.\n', '__package__': '', '__loader__': , '__spec__': , origin='/usr/lib64/python...(truncated), kwdict=) at Python/ceval.c:4641
#44 _PyEval_EvalFrameDefault (
f=Frame 0x7ffff736a3d0, for file , line 219, in _call_with_frames_removed (f=, args=(, {'__name__': 'scandir', '__doc__': 'scandir, a better directory iterator and faster os.walk(), now in the Python 3.5 stdlib\n\nscandir() is a generator version of os.listdir() that returns an\niterator over files in a directory, and also exposes the extra\ninformation most OSes provide while iterating files in a directory\n(such as type and stat
\n\nThis module also includes a version of os.walk() that uses scandir()\nto speed it up significantly.\n\nSee README.md or https://github.com/benhoyt/scandir for rationale and\ndocs, or read PEP 471 (https://www.python.org/dev/peps/pep-0471/) for\nmore details on its inclusion into Python 3.5\n\nscandir is released under the new BSD 3-clause license. See\nLICENSE.txt for the full license text.\n', '__package__': '', '__loader__': ) at Python/ceval.c:3191
#45 0x00005555557da8dd in PyEval_EvalFrameEx (f=, throwflag=0) at Python/ceval.c:547
#46 _PyEval_EvalCodeWithName (_co=, globals=, locals=, args=, argcount=3, kwnames=0x0, kwargs=0x7ffff72ae3c8, kwcount=0, kwstep=1,
defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name='_call_with_frames_removed', qualname='_call_with_frames_removed') at Python/ceval.c:3930
#47 0x000055555573feb8 in _PyFunction_FastCallKeywords (func=, stack=0x21, nargs=0, kwnames=) at Objects/call.c:433
#48 0x00005555557d9b62 in call_function (pp_stack=0x7fffffffabd0, oparg=, kwnames=0x0) at Python/ceval.c:4616
#49 0x00005555557d7002 in _PyEval_EvalFrameDefault (
f=Frame 0x7ffff72ae220, for file , line 728, in exec_module (self=, module=, code=), throwflag=) at Python/ceval.c:3093
#50 0x00005555557403df in function_code_fastcall (co=, args=, nargs=, globals=) at Objects/call.c:283
#51 0x00005555557d9b62 in call_function (pp_stack=0x7fffffffadc0, oparg=, kwnames=0x0) at Python/ceval.c:4616
#52 0x00005555557d6fec in _PyEval_EvalFrameDefault (
f=Frame 0x7ffff73a3de0, for file , line 677, in _load_unlocked (spec=, origin='/usr/lib64/python3.7/site-packages/scandir.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/usr/lib64/python3.7/site-packages/__pycache__/scandir.cpython-37.pyc', _initializing=True) at remote 0x7ffff7367b10>, module=),
throwflag=) at Python/ceval.c:3110
#53 0x00005555557403df in function_code_fastcall (co=, args=, nargs=, globals=) at Objects/call.c:283
#54 0x00005555557d9b62 in call_function (pp_stack=0x7fffffffb010, oparg=, kwnames=0x0) at Python/ceval.c:4616
#55 0x00005555557d708e in _PyEval_EvalFrameDefault (
f=Frame 0x7ffff78be120, for file , line 967, in _find_and_load_unlocked (name='scandir', import_=, path=None, parent='', spec=, origin='/usr/lib64/python3.7/site-packages/scandir.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/usr/lib64/python3.7/site-packages/__pycache__/scandir.cpython-37.pyc', _initializing=True) at remote 0x7ffff7367b10>), throwflag=) at Python/ceval.c:3124
#56 0x00005555557403df in function_code_fastcall (co=, args=, nargs=, globals=) at Objects/call.c:283
#57 0x00005555557d9b62 in call_function (pp_stack=0x7fffffffb200, oparg=, kwnames=0x0) at Python/ceval.c:4616
#58 0x00005555557d708e in _PyEval_EvalFrameDefault (
f=Frame 0x7ffff731a050, for file , line 983, in _find_and_load (name='scandir', import_=, module=), throwflag=) at Python/ceval.c:3124
#59 0x00005555557403df in function_code_fastcall (co=, args=, nargs=, globals=) at Objects/call.c:283
#60 0x0000555555741aae in object_vacall (callable=, vargs=0x7fffffffb360) at Objects/call.c:1202
#61 0x0000555555741b80 in _PyObject_CallMethodIdObjArgs (obj=, name=) at Objects/call.c:1252
#62 0x0000555555802301 in import_find_and_load (abs_name=) at Python/import.c:1648
#63 PyImport_ImportModuleLevelObject (name='scandir', globals=, locals=, fromlist=None, level=0) at Python/import.c:1760
#64 0x00005555557d59eb in import_name (f=, name='scandir', fromlist=None, level=0) at Python/ceval.c:4770
#65 _PyEval_EvalFrameDefault (f=Frame 0x7ffff78baca0, for file , line 1, in (), throwflag=) at Python/ceval.c:2600
#66 0x00005555557da8dd in PyEval_EvalFrameEx (f=, throwflag=0) at Python/ceval.c:547
#67 _PyEval_EvalCodeWithName (_co=, globals=, locals=, args=, argcount=0, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0,
defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:3930
#68 0x00005555557d0cbf in PyEval_EvalCodeEx (_co=, globals=, locals=0x0, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0,
defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3959
#69 PyEval_EvalCode (co=, globals=, locals=0x0) at Python/ceval.c:524
#70 0x000055555581431b in run_mod (mod=, filename=,
globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': },
locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': }, flags=, arena=) at Python/pythonrun.c:1035
#71 PyRun_InteractiveOneObjectEx (fp=, filename=, flags=) at Python/pythonrun.c:256
#72 0x0000555555813afe in PyRun_InteractiveLoopFlags (fp=0x55555a58c6e0, filename_str=, flags=0x7fffffffbad0) at Python/pythonrun.c:120
#73 0x00005555558139f8 in PyRun_AnyFileExFlags (fp=0x55555a58c6e0, filename=0x55555a592ce0 "", closeit=0, flags=0x7fffffffbad0) at Python/pythonrun.c:78
#74 0x00005555556f8b54 in pyembed::pyinterp::MainPythonInterpreter::run_repl (self=0x7fffffffc238) at pyembed/src/pyinterp.rs:739
#75 0x00005555556f57a0 in pyembed::pyinterp::MainPythonInterpreter::run (self=0x7fffffffc238) at pyembed/src/pyinterp.rs:525
#76 0x00005555556f67fc in pyembed::pyinterp::MainPythonInterpreter::run_and_handle_error (self=0x7fffffffc238) at pyembed/src/pyinterp.rs:617
#77 0x00005555556f6b47 in pyembed::pyinterp::MainPythonInterpreter::run_as_main (self=0x7fffffffc238) at pyembed/src/pyinterp.rs:654
#78 0x00005555556bb956 in requests_dec1::main () at src/main.rs:19
```

The backtrace is almost the same for `python_run_mode_eval` and I assume for other run modes.

If it can not be fixed, maybe an `ImportError` can be raised instead.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the crash in repl mode by importing the built scandir extension from sys_paths. Start with the Python import path shown in Python/importdl.c and the PyModule_Create2/PyInit__scandir frames, then isolate why this placement causes the segmentation fault. Done means the extension imports without a segfault in this scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.