`test_embed` segfaults on BOLT instrumented binaries
Chưa có ai nhận issue này.
- Ngôn ngữ chính
- Python
- Star
- 77.2k
- Fork
- 35.9k
- Chỉ số merge pull request
- Chỉ số pull request đang chờ
Mô tả
Bug report
Bug description:
Originally reported in https://github.com/astral-sh/python-build-standalone/pull/463
The test_embed suite fails with BOLT instrumented binaries. This requires configuration with the --enable-bolt and --enable-shared flags as described in #128472.
e.g.
LD_LIBRARY_PATH=/home/zb/workspace/cpython ./python -m test --pgo --timeout=
Using random seed: 703977063
0:00:00 load avg: 4.41 Run 44 tests sequentially in a single process
...
0:00:22 load avg: 3.44 [15/44] test_difflib
0:00:23 load avg: 3.44 [16/44] test_embed
test test_embed failed
0:00:51 load avg: 2.76 [17/44] test_float -- test_embed failed (3 failures)
0:00:51 load avg: 2.76 [18/44] test_fstring
0:00:54 load avg: 2.62 [19/44] test_functools
...
Total duration: 2 min 2 sec
Total tests: run=9,524 failures=3 skipped=219
Total test files: run=44/44 failed=1
Result: FAILURE
I believe this occurs on older Python version too, but is a non-fatal error for the test suite — I have not confirmed this yet. On 3.13+, it's a fatal error.
There's a patch disabling these tests at https://github.com/astral-sh/python-build-standalone/pull/469/commits/50c0317689e18bdb51ae459933366a9752837d17 that I can upstream to unblock BOLT for Python 3.13+. These tests are probably not critical for performance.
Additionally, we'll want to debug why these are segfaulting — that's a bit more involved and can happen after.
Additional logs
Checked 112 modules (34 built-in, 74 shared, 3 n/a on linux-x86_64, 0 disabled, 1 missing, 0 failed on import)
make[2]: Leaving directory '/fast/workspace/cpython'
make[1]: Leaving directory '/fast/workspace/cpython'
make profile-bolt-stamp
make[1]: Entering directory '/fast/workspace/cpython'
# Ensure a pristine, pre-BOLT copy of the binary and no profile data from last run.
for bin in python libpython3.14.so.1.0; do \
prebolt="${bin}.prebolt"; \
if [ -e "${prebolt}" ]; then \
echo "Restoring pre-BOLT binary ${prebolt}"; \
mv "${bin}.prebolt" "${bin}"; \
fi; \
cp "${bin}" "${prebolt}"; \
rm -f ${bin}.bolt.*.fdata ${bin}.fdata; \
done
# Instrument each binary.
for bin in python libpython3.14.so.1.0; do \
/usr/bin/llvm-bolt "${bin}" -instrument -instrumentation-file-append-pid -instrumentation-file=/fast/workspace/cpython/${bin}.bolt -o ${bin}.bolt_inst -update-debug-sections -skip-funcs=_PyEval_EvalFrameDefault,sre_ucs1_match/1,sre_ucs2_match/1,sre_ucs4_match/1 ; \
mv "${bin}.bolt_inst" "${bin}"; \
done
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 6a0964d75628b15bafd078342120888c0e6d126f
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: creating new program header table at address 0x600000, offset 0x200000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling lite mode
BOLT-INFO: 0 out of 6 functions in the binary (0.0%) have non-empty execution profile
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 2
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 4
BOLT-INSTRUMENTER: Number of function descriptors: 4
BOLT-INSTRUMENTER: Number of branch counters: 1
BOLT-INSTRUMENTER: Number of ST leaf node counters: 4
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 5
BOLT-INSTRUMENTER: Total size of counters: 40 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 47 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 356 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /fast/workspace/cpython/python.bolt
BOLT-INFO: padding code to 0xa00000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0xa06950
BOLT-INFO: clear procedure is 0xa02390
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting _end to 0xa0a0bc
BOLT-INFO: setting _end to 0xa0a0bc
BOLT-INFO: setting __bolt_runtime_start to 0xa06900
BOLT-INFO: setting __bolt_runtime_fini to 0xa06950
BOLT-INFO: setting __hot_start to 0x800000
BOLT-INFO: setting __hot_end to 0x80014c
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 6a0964d75628b15bafd078342120888c0e6d126f
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x600000, offset 0x600000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling lite mode
BOLT-WARNING: Failed to analyze 1380 relocations
BOLT-INFO: 0 out of 7363 functions in the binary (0.0%) have non-empty execution profile
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 1508
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 7281
BOLT-INSTRUMENTER: Number of function descriptors: 7281
BOLT-INSTRUMENTER: Number of branch counters: 82309
BOLT-INSTRUMENTER: Number of ST leaf node counters: 43949
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 126258
BOLT-INSTRUMENTER: Total size of counters: 1010064 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 186842 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 8303240 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /fast/workspace/cpython/libpython3.14.so.1.0.bolt
BOLT-INFO: 59587 instructions were shortened
BOLT-INFO: removed 161 empty blocks
BOLT-INFO: UCE removed 1623 blocks and 100292 bytes of code
BOLT-INFO: padding code to 0x1000000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0x11bd950
BOLT-INFO: clear procedure is 0x11b9390
BOLT-INFO: patched build-id (flipped last bit)
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-WARNING: [internal-dwarf-error]: unsupported DW_OP encoding.
BOLT-INFO: setting __bolt_runtime_start to 0x11bd900
BOLT-INFO: setting __bolt_runtime_fini to 0x11bd950
BOLT-INFO: setting __hot_start to 0x800000
BOLT-INFO: setting __hot_end to 0xe06596
# Run instrumented binaries to collect data.
LD_LIBRARY_PATH=/home/zb/workspace/cpython ./python -m test --pgo --timeout=
Using random seed: 703977063
0:00:00 load avg: 4.41 Run 44 tests sequentially in a single process
0:00:00 load avg: 4.41 [ 1/44] test_array
0:00:01 load avg: 4.41 [ 2/44] test_base64
0:00:03 load avg: 4.41 [ 3/44] test_binascii
0:00:04 load avg: 4.41 [ 4/44] test_binop
0:00:04 load avg: 4.41 [ 5/44] test_bisect
0:00:04 load avg: 4.41 [ 6/44] test_bytes
0:00:08 load avg: 4.14 [ 7/44] test_bz2
0:00:09 load avg: 3.89 [ 8/44] test_cmath
0:00:09 load avg: 3.89 [ 9/44] test_codecs
0:00:11 load avg: 3.89 [10/44] test_collections
0:00:12 load avg: 3.89 [11/44] test_complex
0:00:13 load avg: 3.89 [12/44] test_dataclasses
0:00:13 load avg: 3.89 [13/44] test_datetime
0:00:19 load avg: 3.65 [14/44] test_decimal
0:00:22 load avg: 3.44 [15/44] test_difflib
0:00:23 load avg: 3.44 [16/44] test_embed
test test_embed failed
0:00:51 load avg: 2.76 [17/44] test_float -- test_embed failed (3 failures)
0:00:51 load avg: 2.76 [18/44] test_fstring
0:00:54 load avg: 2.62 [19/44] test_functools
0:00:56 load avg: 2.62 [20/44] test_generators
0:00:57 load avg: 2.62 [21/44] test_hashlib
0:00:57 load avg: 2.62 [22/44] test_heapq
0:00:58 load avg: 2.62 [23/44] test_int
0:01:00 load avg: 2.49 [24/44] test_itertools
0:01:05 load avg: 2.37 [25/44] test_json
0:01:18 load avg: 2.47 [26/44] test_long
0:01:20 load avg: 2.35 [27/44] test_lzma
0:01:21 load avg: 2.35 [28/44] test_math
0:01:24 load avg: 2.35 [29/44] test_memoryview
0:01:25 load avg: 2.24 [30/44] test_operator
0:01:25 load avg: 2.24 [31/44] test_ordered_dict
0:01:28 load avg: 2.24 [32/44] test_patma
0:01:29 load avg: 2.24 [33/44] test_pickle
0:01:34 load avg: 2.14 [34/44] test_pprint
0:01:34 load avg: 2.05 [35/44] test_re
0:01:35 load avg: 2.05 [36/44] test_set
0:01:41 load avg: 1.97 [37/44] test_sqlite3
0:01:44 load avg: 1.97 [38/44] test_statistics
0:01:49 load avg: 1.89 [39/44] test_str
0:01:52 load avg: 1.82 [40/44] test_struct
0:01:53 load avg: 1.82 [41/44] test_tabnanny
0:01:56 load avg: 1.91 [42/44] test_time
0:01:58 load avg: 1.91 [43/44] test_xml_etree
0:02:00 load avg: 1.84 [44/44] test_xml_etree_c
Total duration: 2 min 2 sec
Total tests: run=9,524 failures=3 skipped=219
Total test files: run=44/44 failed=1
Result: FAILURE
make[1]: *** [Makefile:969: profile-bolt-stamp] Error 2
make[1]: Leaving directory '/fast/workspace/cpython'
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
- gh-128474
Hướng dẫn đóng góp
Bắt đầu từ đâu
- Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
- Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
- Fork repository và làm thay đổi trên một nhánh.
- Mở pull request có tham chiếu số hiệu của issue.
Hướng nghiên cứu
Xem xét gh-128474 và các lỗi của test_embed khi dùng --enable-bolt và --enable-shared. Tái hiện bằng lệnh LD_LIBRARY_PATH=... ./python -m test --pgo --timeout= được ghi trong tài liệu, sau đó kiểm tra test_embed suite và ba lỗi của nó. Hoàn tất khi lần chạy kiểm thử được instrument bằng BOLT không còn bị segfault hoặc có cách xử lý đã được thống nhất và kiểm thử cho các kiểm thử này.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- python
- Lĩnh vực
- build-system, testing-qa
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Cần làm rõ
- Mức phù hợp với người mới
- 25/100