python / python/cpython

`_remote_debugging`: binary format only `total_samples:u32`

Offen
#151,292 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

extension-modules topic-profiling type-bug
Vorherrschende Sprache
Python
Sterne
77.2k
Forks
35.9k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

Bug report

Bug description:

The binary format defines total_samples as just u32:

https://github.com/python/cpython/blob/540b3d0a7fa7cd842f064f79b1410cbd6868bffa/Modules/_remote_debugging/binary_io.h#L269

https://github.com/python/cpython/blob/540b3d0a7fa7cd842f064f79b1410cbd6868bffa/Modules/_remote_debugging/binary_io.h#L53-L54

That's not that much... With just 100khz:

Threads overflow after...
1 ~11.9 h
4 ~3.0 h
10 ~71 min
64 ~11 min

especially if we aim for continous profiling of real production systems. But even of macOS, I'm observing ~0.5-1MHz on my mach_vm_remap branch already...

To make matters worse, since #150349 it results in OverflowError and the binary file is just corrupted:

2026-06-11T02:03:53.357502000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (vmremap 0bdde7f?) % ls -l /tmp/overflow.bin
-rw-r--r--@ 1 root  wheel  4894411257 Jun 11 01:45 /tmp/overflow.bin
2026-06-11T02:03:58.920689000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (vmremap 0bdde7f?) % head -c 128 /tmp/overflow.bin | xxd
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000040: 28b5 2ffd 0058 5c1d 013a d500 1621 8025  (./..X\..:...!.%
00000050: 49d2 0133 3003 331a 3330 3d32 35a9 ccc0  I..30.3.30=25...
00000060: 3aa9 049c b5d6 b69d d296 3122 4488 0c6d  :.........1"D..m
00000070: 0148 0150 014f 0946 32bd b931 318c b6de  .H.P.O.F2..11...

as the header is written on finalize:

https://github.com/python/cpython/blob/540b3d0a7fa7cd842f064f79b1410cbd6868bffa/Modules/_remote_debugging/binary_io_writer.c#L1074
https://github.com/python/cpython/blob/540b3d0a7fa7cd842f064f79b1410cbd6868bffa/Modules/_remote_debugging/binary_io_writer.c#L1196-L1223

To put aside that u32 is just too little, we should be graceful here.

Other fields seem to be fine, but need a double check here. I think that rotating files is a stop gap, and we need chunking.

Reproduction

(as a part of routine stress tests of https://github.com/maurycy/cpython/pull/3)

[130] 2026-06-10T23:35:58.225891000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (vmremap 92ad857*?) % sudo ./python.exe -m profiling.sampling run --binary -r 1000khz -d 25000 -o /tmp/overflow.bin --realtime-stats busywork.py
Stats: 647,833.9Hz (1.5µs) Min: 615,379.9Hz Max: 705,712.7Hz N=4294631818 Cache: 100.0% (4294631818+0/1)Traceback (most recent call last):
  File "<frozen runpy>", line 201, in _run_module_as_main
  File "<frozen runpy>", line 87, in _run_code
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/__main__.py", line 65, in <module>
    main()
    ~~~~^^
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/cli.py", line 977, in main
    _main()
    ~~~~~^^
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/cli.py", line 1133, in _main
    handler(args)
    ~~~~~~~^^^^^^
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/cli.py", line 1280, in _handle_run
    collector = sample(
        process.pid,
    ...<9 lines>...
        blocking=args.blocking,
    )
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/sample.py", line 504, in sample
    profiler.sample(collector, duration_sec, async_aware=async_aware)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/sample.py", line 167, in sample
    raise e from None
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/sample.py", line 155, in sample
    collector.collect(stack_frames)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/binary_collector.py", line 84, in collect
    self._writer.write_sample(stack_frames, timestamp_us)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OverflowError: too many samples for binary format
[1] 2026-06-11T01:47:42.249903000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (vmremap cd5a675*?) %
def hot_a(n):
    return sum(i * i for i in range(n))


def hot_b(n):
    return sum(i + i for i in range(n))


def worker():
    while True:
        hot_a(6_000_000)
        hot_b(6_000_000)


worker()
CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs
  • gh-152892
  • gh-153425
  • gh-153436

Beitragsleitfaden

Beitragsleitfaden öffnen

Erste Schritte

  1. Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
  3. Forke das Repository und arbeite in einem Branch.
  4. Öffne einen Pull Request, der die Issue-Nummer nennt.

Rechercherichtung

Beginne mit Modules/_remote_debugging/binary_io.h und binary_io_writer.c, und untersuche anschließend Lib/profiling/sampling/binary_collector.py sowie die verknüpften PRs gh-152892, gh-153425 und gh-153436. Reproduziere das Problem mit dem bereitgestellten Profiling-Befehl und überprüfe, dass Sampling mit hoher Rate nun weder OverflowError auslöst noch eine beschädigte Binärausgabe hinterlässt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
c, python
Bereich
observability, performance
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.