python / python/cpython

`test_embed` segfaults on BOLT instrumented binaries

Open
#128,473 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

build tests type-crash
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR metrics pending

Description

Bug report

Bug description:

Originally reported in https://github.com/astral-sh/python-build-standalone/pull/463

The test_embed suite fails with BOLT instrumented binaries. This requires configuration with the --enable-bolt and --enable-shared flags as described in #128472.

e.g.

LD_LIBRARY_PATH=/home/zb/workspace/cpython ./python -m test --pgo --timeout=
Using random seed: 703977063
0:00:00 load avg: 4.41 Run 44 tests sequentially in a single process
...
0:00:22 load avg: 3.44 [15/44] test_difflib
0:00:23 load avg: 3.44 [16/44] test_embed
test test_embed failed
0:00:51 load avg: 2.76 [17/44] test_float -- test_embed failed (3 failures)
0:00:51 load avg: 2.76 [18/44] test_fstring
0:00:54 load avg: 2.62 [19/44] test_functools
...
Total duration: 2 min 2 sec
Total tests: run=9,524 failures=3 skipped=219
Total test files: run=44/44 failed=1
Result: FAILURE

I believe this occurs on older Python version too, but is a non-fatal error for the test suite — I have not confirmed this yet. On 3.13+, it's a fatal error.

There's a patch disabling these tests at https://github.com/astral-sh/python-build-standalone/pull/469/commits/50c0317689e18bdb51ae459933366a9752837d17 that I can upstream to unblock BOLT for Python 3.13+. These tests are probably not critical for performance.

Additionally, we'll want to debug why these are segfaulting — that's a bit more involved and can happen after.

Additional logs 
Checked 112 modules (34 built-in, 74 shared, 3 n/a on linux-x86_64, 0 disabled, 1 missing, 0 failed on import)
make[2]: Leaving directory '/fast/workspace/cpython'
make[1]: Leaving directory '/fast/workspace/cpython'
make profile-bolt-stamp
make[1]: Entering directory '/fast/workspace/cpython'
# Ensure a pristine, pre-BOLT copy of the binary and no profile data from last run.
for bin in python libpython3.14.so.1.0; do \
  prebolt="${bin}.prebolt"; \
  if [ -e "${prebolt}" ]; then \
    echo "Restoring pre-BOLT binary ${prebolt}"; \
    mv "${bin}.prebolt" "${bin}"; \
  fi; \
  cp "${bin}" "${prebolt}"; \
  rm -f ${bin}.bolt.*.fdata ${bin}.fdata; \
done
# Instrument each binary.
for bin in python libpython3.14.so.1.0; do \
  /usr/bin/llvm-bolt "${bin}" -instrument -instrumentation-file-append-pid -instrumentation-file=/fast/workspace/cpython/${bin}.bolt -o ${bin}.bolt_inst -update-debug-sections -skip-funcs=_PyEval_EvalFrameDefault,sre_ucs1_match/1,sre_ucs2_match/1,sre_ucs4_match/1 ; \
  mv "${bin}.bolt_inst" "${bin}"; \
done
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 6a0964d75628b15bafd078342120888c0e6d126f
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: creating new program header table at address 0x600000, offset 0x200000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling lite mode
BOLT-INFO: 0 out of 6 functions in the binary (0.0%) have non-empty execution profile
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 2
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 4
BOLT-INSTRUMENTER: Number of function descriptors: 4
BOLT-INSTRUMENTER: Number of branch counters: 1
BOLT-INSTRUMENTER: Number of ST leaf node counters: 4
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 5
BOLT-INSTRUMENTER: Total size of counters: 40 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 47 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 356 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /fast/workspace/cpython/python.bolt
BOLT-INFO: padding code to 0xa00000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0xa06950
BOLT-INFO: clear procedure is 0xa02390
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting _end to 0xa0a0bc
BOLT-INFO: setting _end to 0xa0a0bc
BOLT-INFO: setting __bolt_runtime_start to 0xa06900
BOLT-INFO: setting __bolt_runtime_fini to 0xa06950
BOLT-INFO: setting __hot_start to 0x800000
BOLT-INFO: setting __hot_end to 0x80014c
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 6a0964d75628b15bafd078342120888c0e6d126f
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x600000, offset 0x600000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling lite mode
BOLT-WARNING: Failed to analyze 1380 relocations
BOLT-INFO: 0 out of 7363 functions in the binary (0.0%) have non-empty execution profile
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 1508
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 7281
BOLT-INSTRUMENTER: Number of function descriptors: 7281
BOLT-INSTRUMENTER: Number of branch counters: 82309
BOLT-INSTRUMENTER: Number of ST leaf node counters: 43949
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 126258
BOLT-INSTRUMENTER: Total size of counters: 1010064 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 186842 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 8303240 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /fast/workspace/cpython/libpython3.14.so.1.0.bolt
BOLT-INFO: 59587 instructions were shortened
BOLT-INFO: removed 161 empty blocks
BOLT-INFO: UCE removed 1623 blocks and 100292 bytes of code
BOLT-INFO: padding code to 0x1000000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0x11bd950
BOLT-INFO: clear procedure is 0x11b9390
BOLT-INFO: patched build-id (flipped last bit)
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-INFO: setting __bolt_runtime_start to 0x11bd900
BOLT-INFO: setting __bolt_runtime_fini to 0x11bd950
BOLT-INFO: setting __hot_start to 0x800000
BOLT-INFO: setting __hot_end to 0xe06596
# Run instrumented binaries to collect data.
LD_LIBRARY_PATH=/home/zb/workspace/cpython ./python -m test --pgo --timeout=
Using random seed: 703977063
0:00:00 load avg: 4.41 Run 44 tests sequentially in a single process
0:00:00 load avg: 4.41 [ 1/44] test_array
0:00:01 load avg: 4.41 [ 2/44] test_base64
0:00:03 load avg: 4.41 [ 3/44] test_binascii
0:00:04 load avg: 4.41 [ 4/44] test_binop
0:00:04 load avg: 4.41 [ 5/44] test_bisect
0:00:04 load avg: 4.41 [ 6/44] test_bytes
0:00:08 load avg: 4.14 [ 7/44] test_bz2
0:00:09 load avg: 3.89 [ 8/44] test_cmath
0:00:09 load avg: 3.89 [ 9/44] test_codecs
0:00:11 load avg: 3.89 [10/44] test_collections
0:00:12 load avg: 3.89 [11/44] test_complex
0:00:13 load avg: 3.89 [12/44] test_dataclasses
0:00:13 load avg: 3.89 [13/44] test_datetime
0:00:19 load avg: 3.65 [14/44] test_decimal
0:00:22 load avg: 3.44 [15/44] test_difflib
0:00:23 load avg: 3.44 [16/44] test_embed
test test_embed failed
0:00:51 load avg: 2.76 [17/44] test_float -- test_embed failed (3 failures)
0:00:51 load avg: 2.76 [18/44] test_fstring
0:00:54 load avg: 2.62 [19/44] test_functools
0:00:56 load avg: 2.62 [20/44] test_generators
0:00:57 load avg: 2.62 [21/44] test_hashlib
0:00:57 load avg: 2.62 [22/44] test_heapq
0:00:58 load avg: 2.62 [23/44] test_int
0:01:00 load avg: 2.49 [24/44] test_itertools
0:01:05 load avg: 2.37 [25/44] test_json
0:01:18 load avg: 2.47 [26/44] test_long
0:01:20 load avg: 2.35 [27/44] test_lzma
0:01:21 load avg: 2.35 [28/44] test_math
0:01:24 load avg: 2.35 [29/44] test_memoryview
0:01:25 load avg: 2.24 [30/44] test_operator
0:01:25 load avg: 2.24 [31/44] test_ordered_dict
0:01:28 load avg: 2.24 [32/44] test_patma
0:01:29 load avg: 2.24 [33/44] test_pickle
0:01:34 load avg: 2.14 [34/44] test_pprint
0:01:34 load avg: 2.05 [35/44] test_re
0:01:35 load avg: 2.05 [36/44] test_set
0:01:41 load avg: 1.97 [37/44] test_sqlite3
0:01:44 load avg: 1.97 [38/44] test_statistics
0:01:49 load avg: 1.89 [39/44] test_str
0:01:52 load avg: 1.82 [40/44] test_struct
0:01:53 load avg: 1.82 [41/44] test_tabnanny
0:01:56 load avg: 1.91 [42/44] test_time
0:01:58 load avg: 1.91 [43/44] test_xml_etree
0:02:00 load avg: 1.84 [44/44] test_xml_etree_c
Total duration: 2 min 2 sec
Total tests: run=9,524 failures=3 skipped=219
Total test files: run=44/44 failed=1
Result: FAILURE
make[1]: *** [Makefile:969: profile-bolt-stamp] Error 2
make[1]: Leaving directory '/fast/workspace/cpython'
CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs
  • gh-128474

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review gh-128474 and the test_embed failures under --enable-bolt and --enable-shared. Reproduce with the documented LD_LIBRARY_PATH=... ./python -m test --pgo --timeout= command, then inspect the test_embed suite and its three failures. Done means the BOLT-instrumented test run no longer segfaults or has an agreed, tested handling for these tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.