rust-lang / rust-lang/rust

Segmentation fault in `__llvm_write_binary_ids` using LLVM 17.0.6

Open
#123,361 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

A-LLVM A-profile-rt C-bug S-needs-repro
Dominant language
Rust
Stars
119k
Forks
16.1k
PR metrics pending

Description

I have a Python binding using pyo3. In my GitHub Actions pipeline, I gather PGO data. After upgrading from v1.76.0 to v1.77.1 my pipeline gives segfault.

I expected to see this happen: prof data should be saved correctly

Instead, this happened: SIGSEGV

I am so sorry for not providing the code for reproducing and clear details. I wanna share the backtrace with relative people who can guess or have already seen such an error. My notes:

  • I can't reproduce it locally. Affects only GitHub Runners. Probably because they are in the Docker?
  • It raises segfault only when I set RUSTFLAGS="-Cprofile-generate=profdata"
  • It can't be reproducible with debug=True
  • It can't be reproducible with opt-level=2
  • In so rare cases CI could pass...
  • I found these changes in __llvm_write_binary_ids between LLVM 1.16 and 1.17, Which originally comes from https://reviews.llvm.org/D146976
  • The original issue where CI fails is here
Meta

rustc --version --verbose:

rustc 1.77.1 (7cf61ebde 2024-03-27)
binary: rustc
commit-hash: 7cf61ebde7b22796c69757901dd346d0fe70bd97
commit-date: 2024-03-27
host: x86_64-unknown-linux-gnu
release: 1.77.1
LLVM version: 17.0.6
Backtrace 

Reading symbols from /opt/hostedtoolcache/Python/3.12.2/x64/bin/python3.12...
[New LWP 12989]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/hostedtoolcache/Python/3.12.2/x64/bin/python /opt/hostedtoolcache/Python/3'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007effeaeedba5 in __llvm_write_binary_ids () from /opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/libipld/libipld.cpython-312-x86_64-linux-gnu.so
(gdb) bt
#0  0x00007effeaeedba5 in __llvm_write_binary_ids () from /opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/libipld/libipld.cpython-312-x86_64-linux-gnu.so
#1  0x00007effeaeee54f in lprofWriteDataImpl () from /opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/libipld/libipld.cpython-312-x86_64-linux-gnu.so
#2  0x00007effeaeee42c in lprofWriteData () from /opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/libipld/libipld.cpython-312-x86_64-linux-gnu.so
#3  0x00007effeaeec799 in writeFile () from /opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/libipld/libipld.cpython-312-x86_64-linux-gnu.so
#4  0x00007effeaeec512 in __llvm_profile_write_file () from /opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/libipld/libipld.cpython-312-x86_64-linux-gnu.so
#5  0x00007effecc45495 in __run_exit_handlers (status=0, listp=0x7effece1a838 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:113
#6  0x00007effecc45610 in __GI_exit (status=<optimized out>) at ./stdlib/exit.c:143
#7  0x00007effed3299ab in Py_Exit (sts=0) at Python/pylifecycle.c:3060
#8  0x00007effed32bcab in handle_system_exit () at Python/pythonrun.c:756
#9  0x00007effed32baf8 in _PyErr_PrintEx (tstate=0x7effed6ab648 <_PyRuntime+459656>, set_sys_last_vars=1) at Python/pythonrun.c:765
#10 0x00007effed1b5402 in PyErr_PrintEx (set_sys_last_vars=1) at Python/pythonrun.c:845
#11 PyErr_Print () at Python/pythonrun.c:851
#12 _PyRun_SimpleFileObject (fp=<optimized out>, filename=<optimized out>, closeit=1, flags=0x7ffeed8807f0) at Python/pythonrun.c:439
#13 0x00007effed32b6c7 in _PyRun_AnyFileObject (fp=fp@entry=0x55cf58114490, filename=filename@entry=0x7effecb2f870, closeit=closeit@entry=1, flags=flags@entry=0x7ffeed8807f0) at Python/pythonrun.c:78
#14 0x00007effed3343e8 in pymain_run_file_obj (skip_source_first_line=<optimized out>, filename=0x7effecb2f870, program_name=0x7effecb2f810) at Modules/main.c:360
#15 pymain_run_file (config=0x7effed64e228 <_PyRuntime+77672>) at Modules/main.c:379
#16 pymain_run_python (exitcode=0x7ffeed8807c4) at Modules/main.c:629
#17 Py_RunMain () at Modules/main.c:709
#18 0x00007effed333f4d in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:763
#19 0x00007effecc29d90 in __libc_start_call_main (main=main@entry=0x55cf56838060 <main>, argc=argc@entry=3, argv=argv@entry=0x7ffeed880a48) at ../sysdeps/nptl/libc_start_call_main.h:58
#20 0x00007effecc29e40 in __libc_start_main_impl (main=0x55cf56838060 <main>, argc=3, argv=0x7ffeed880a48, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffeed880a38) at ../csu/libc-start.c:392
#21 0x000055cf56838095 in _start ()

Summary:

           PID: 12989 (pytest)
           UID: 1001 (runner)
           GID: 127 (docker)
        Signal: 11 (SEGV)
     Timestamp: Mon 2024-04-01 20:45:59 UTC (2min 35s ago)
  Command Line: /opt/hostedtoolcache/Python/3.12.2/x64/bin/python /opt/hostedtoolcache/Python/3.12.2/x64/bin/pytest .
    Executable: /opt/hostedtoolcache/Python/3.12.2/x64/bin/python3.12
 Control Group: /system.slice/runner-provisioner.service
          Unit: runner-provisioner.service
         Slice: system.slice
       Boot ID: 5b47de7d31be4ba3ac7a1d4bfccc60c9
    Machine ID: 87c9cb22384e417dbad323c8a7783269
      Hostname: fv-az979-578
       Storage: /var/lib/systemd/coredump/core.pytest.1001.5b47de7d31be4ba3ac7a1d4bfccc60c9.12989.1712004359000000.zst (present)
     Disk Size: 6.9M
       Message: Process 12989 (pytest) of user 1001 dumped core.

                Found module linux-vdso.so.1 with build-id: 151d9b890f2a92eb30984bdf8c8d264337b7b456
                Found module libgcc_s.so.1 with build-id: 12424bb285e85ee967d17e4f87e4106889688eb0
                Found module libipld.cpython-312-x86_64-linux-gnu.so with build-id: 43dc019f23e4f2fe07b7bbaf058486b239c29ac2
                Found module libssl.so.3 with build-id: 9e2ad2d446d1e0e442154f9df9a3daf6a04bd645
                Found module _ssl.cpython-312-x86_64-linux-gnu.so with build-id: 450f823f91c0465be9a47c253c91afe5ae848168
                Found module _statistics.cpython-312-x86_64-linux-gnu.so with build-id: 3df44b2f27d3c7422a609431b5a7344bd7c3f886
                Found module _lsprof.cpython-312-x86_64-linux-gnu.so with build-id: ca6e3d25651783d61b6f7c76a04f163ac5be2fb4
                Found module _queue.cpython-312-x86_64-linux-gnu.so with build-id: e981ca495370f5ac867b7d5f8a49047034edad6c
                Found module _blake2.cpython-312-x86_64-linux-gnu.so with build-id: f308a79989edd16bff485378362a7b422e46fe60
                Found module libcrypto.so.3 with build-id: ca84e22a798dabf117de600cb13469a59f775d2a
                Found module _hashlib.cpython-312-x86_64-linux-gnu.so with build-id: 8984f3ca2381a3cbb369f3c7aeb8a9b3122bf6ba
                Found module pyexpat.cpython-312-x86_64-linux-gnu.so with build-id: bdd1aac422932880036e73e05e1d39655b586095
                Found module _elementtree.cpython-312-x86_64-linux-gnu.so with build-id: 0cf6d36b147ecd083fdedfe5d223be5597b9b30a
                Found module _posixsubprocess.cpython-312-x86_64-linux-gnu.so with build-id: c0fb0a4e59feaae1956db1a3c7d903e9110e45b8
                Found module fcntl.cpython-312-x86_64-linux-gnu.so with build-id: b651a042c8c2c3cdf32921f7dbbbd534184d1941
                Found module _decimal.cpython-312-x86_64-linux-gnu.so with build-id: 2c274387bfdebc25573c1ac2c460a76edadad83e
                Found module _heapq.cpython-312-x86_64-linux-gnu.so with build-id: d88205451b958a9c655ed5ef36b9aff8085b9505
                Found module _json.cpython-312-x86_64-linux-gnu.so with build-id: f85a045dd58f1354cb1d2d787511bb54f65cfbe7
                Found module _datetime.cpython-312-x86_64-linux-gnu.so with build-id: 9b9e29e510192e8bf237f2b1c0322a3bc78b0b1b
                Found module array.cpython-312-x86_64-linux-gnu.so with build-id: d5be42f2fde84d729406e16b01800aedca0bb38c
                Found module select.cpython-312-x86_64-linux-gnu.so with build-id: d9cb78bacb0d6ea54bbf2e4e614a1a122f0c8ee2
                Found module _socket.cpython-312-x86_64-linux-gnu.so with build-id: ee6916c3154f6c453c8a1419cebdfa0c01c9b65d
                Found module _sha2.cpython-312-x86_64-linux-gnu.so with build-id: 502fc65c7626e053c47d67169d58ad116885ef9f
                Found module _random.cpython-312-x86_64-linux-gnu.so with build-id: 3b62fa7ba651f2ee03148ea4c73a898bcfcbcf24
                Found module _struct.cpython-312-x86_64-linux-gnu.so with build-id: c64dcced7af8a3aa224d54cde5c4b9f07084637b
                Found module binascii.cpython-312-x86_64-linux-gnu.so with build-id: 2ea43e93b90075ab910aa5efe465f7248068dd9e
                Found module _csv.cpython-312-x86_64-linux-gnu.so with build-id: c9a54841ae7c5c8bdcb62b46b308ed3bc88605b6
                Found module libuuid.so.1 with build-id: 64c0d0cb22fa2bdeca075a0c0418ba5ff314b220
                Found module _uuid.cpython-312-x86_64-linux-gnu.so with build-id: cc667838deb0fa7cd4ca5c23173c377411360452
                Found module unicodedata.cpython-312-x86_64-linux-gnu.so with build-id: dcdfe492981da31e3f725d4e08848754edfe88bd
                Found module liblzma.so.5 with build-id: b85da6c48eb60a646615392559483b93617ef265
                Found module _lzma.cpython-312-x86_64-linux-gnu.so with build-id: 5c4273615de20517bea244ca4440675b311804a2
                Found module libbz2.so.1.0 with build-id: e56b62c27bcc7ace8f9be36b255bd7b31bfde405
                Found module _bz2.cpython-312-x86_64-linux-gnu.so with build-id: cbb7f493ef63c0e1b17e306b860badaa2919056c
                Found module libz.so.1 with build-id: 30840b79ac329ecbf1dec0bb60180eed256d319f
                Found module zlib.cpython-312-x86_64-linux-gnu.so with build-id: 764c9f059514c4616d487786a646236f5a101de4
                Found module _bisect.cpython-312-x86_64-linux-gnu.so with build-id: c70b6223bdefdfcd8bb4fb6b1fe4ef4ac06fb44f
                Found module math.cpython-312-x86_64-linux-gnu.so with build-id: bb6fc1ebbfb4ffc9e3e37fd2eb696bc29fe3a6f7
                Found module _opcode.cpython-312-x86_64-linux-gnu.so with build-id: b2b598c52d5048d7d5bfd6ef276047e65d430b4e
                Found module ld-linux-x86-64.so.2 with build-id: 15921ea631d9f36502d20459c43e5c85b7d6ab76
                Found module libm.so.6 with build-id: a88ef0199bd5e742ebd0c359edf5cb2be0ec08b5
                Found module libc.so.6 with build-id: c289da5071a3399de893d2af81d6a30c62646e1e
                Found module libpython3.12.so.1.0 with build-id: 6c2f4f5468d6052d6cfbbbc81c360981a2248326
                Found module python3.12 with build-id: f4f4bbd3f484b21b9ba4ea26f7c7139b77fc2f12
                Stack trace of thread 12989:
                #0  0x00007effeaeedba5 __llvm_write_binary_ids (libipld.cpython-312-x86_64-linux-gnu.so + 0xedba5)
                #1  0x00007effeaeee54f lprofWriteDataImpl (libipld.cpython-312-x86_64-linux-gnu.so + 0xee54f)
                #2  0x00007effeaeee42c lprofWriteData (libipld.cpython-312-x86_64-linux-gnu.so + 0xee42c)
                #3  0x00007effeaeec799 writeFile (libipld.cpython-312-x86_64-linux-gnu.so + 0xec799)
                #4  0x00007effeaeec512 __llvm_profile_write_file (libipld.cpython-312-x86_64-linux-gnu.so + 0xec512)
                #5  0x00007effecc45495 __run_exit_handlers (libc.so.6 + 0x45495)
                #6  0x00007effecc45610 __GI_exit (libc.so.6 + 0x45610)
                #7  0x00007effed3299ab Py_Exit (libpython3.12.so.1.0 + 0x3299ab)
                #8  0x00007effed32bcab handle_system_exit (libpython3.12.so.1.0 + 0x32bcab)
                #9  0x00007effed32baf8 _PyErr_PrintEx (libpython3.12.so.1.0 + 0x32baf8)
                #10 0x00007effed1b5402 PyErr_PrintEx (libpython3.12.so.1.0 + 0x1b5402)
                #11 0x00007effed32b6c7 _PyRun_AnyFileObject (libpython3.12.so.1.0 + 0x32b6c7)
                #12 0x00007effed3343e8 pymain_run_file_obj (libpython3.12.so.1.0 + 0x3343e8)
                #13 0x00007effed333f4d Py_BytesMain (libpython3.12.so.1.0 + 0x333f4d)
                #14 0x00007effecc29d90 __libc_start_call_main (libc.so.6 + 0x29d90)
                #15 0x00007effecc29e40 __libc_start_main_impl (libc.so.6 + 0x29e40)
                #16 0x000055cf56838095 _start (python3.12 + 0x1095)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported rustc 1.77.1/LLVM 17.0.6 backtrace at __llvm_write_binary_ids and the RUSTFLAGS=-Cprofile-generate=profdata configuration, comparing the 1.76.0 and 1.77.1 behavior in GitHub Actions. A useful result would reproduce the SIGSEGV or narrow its triggering conditions and verify that PGO data is saved correctly without the crash.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python, rust
Domain
ci-cd, compilers
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.