python / python/cpython

`_remote_debugging`: binary format only `total_samples:u32`

Open
#151,292 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

extension-modules topic-profiling type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

The binary format defines total_samples as just u32:

https://github.com/python/cpython/blob/540b3d0a7fa7cd842f064f79b1410cbd6868bffa/Modules/_remote_debugging/binary_io.h#L269

https://github.com/python/cpython/blob/540b3d0a7fa7cd842f064f79b1410cbd6868bffa/Modules/_remote_debugging/binary_io.h#L53-L54

That's not that much... With just 100khz:

Threads overflow after...
1 ~11.9 h
4 ~3.0 h
10 ~71 min
64 ~11 min

especially if we aim for continous profiling of real production systems. But even of macOS, I'm observing ~0.5-1MHz on my mach_vm_remap branch already...

To make matters worse, since #150349 it results in OverflowError and the binary file is just corrupted:

2026-06-11T02:03:53.357502000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (vmremap 0bdde7f?) % ls -l /tmp/overflow.bin
-rw-r--r--@ 1 root  wheel  4894411257 Jun 11 01:45 /tmp/overflow.bin
2026-06-11T02:03:58.920689000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (vmremap 0bdde7f?) % head -c 128 /tmp/overflow.bin | xxd
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000040: 28b5 2ffd 0058 5c1d 013a d500 1621 8025  (./..X\..:...!.%
00000050: 49d2 0133 3003 331a 3330 3d32 35a9 ccc0  I..30.3.30=25...
00000060: 3aa9 049c b5d6 b69d d296 3122 4488 0c6d  :.........1"D..m
00000070: 0148 0150 014f 0946 32bd b931 318c b6de  .H.P.O.F2..11...

as the header is written on finalize:

https://github.com/python/cpython/blob/540b3d0a7fa7cd842f064f79b1410cbd6868bffa/Modules/_remote_debugging/binary_io_writer.c#L1074
https://github.com/python/cpython/blob/540b3d0a7fa7cd842f064f79b1410cbd6868bffa/Modules/_remote_debugging/binary_io_writer.c#L1196-L1223

To put aside that u32 is just too little, we should be graceful here.

Other fields seem to be fine, but need a double check here. I think that rotating files is a stop gap, and we need chunking.

Reproduction

(as a part of routine stress tests of https://github.com/maurycy/cpython/pull/3)

[130] 2026-06-10T23:35:58.225891000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (vmremap 92ad857*?) % sudo ./python.exe -m profiling.sampling run --binary -r 1000khz -d 25000 -o /tmp/overflow.bin --realtime-stats busywork.py
Stats: 647,833.9Hz (1.5µs) Min: 615,379.9Hz Max: 705,712.7Hz N=4294631818 Cache: 100.0% (4294631818+0/1)Traceback (most recent call last):
  File "<frozen runpy>", line 201, in _run_module_as_main
  File "<frozen runpy>", line 87, in _run_code
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/__main__.py", line 65, in <module>
    main()
    ~~~~^^
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/cli.py", line 977, in main
    _main()
    ~~~~~^^
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/cli.py", line 1133, in _main
    handler(args)
    ~~~~~~~^^^^^^
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/cli.py", line 1280, in _handle_run
    collector = sample(
        process.pid,
    ...<9 lines>...
        blocking=args.blocking,
    )
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/sample.py", line 504, in sample
    profiler.sample(collector, duration_sec, async_aware=async_aware)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/sample.py", line 167, in sample
    raise e from None
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/sample.py", line 155, in sample
    collector.collect(stack_frames)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/profiling/sampling/binary_collector.py", line 84, in collect
    self._writer.write_sample(stack_frames, timestamp_us)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OverflowError: too many samples for binary format
[1] 2026-06-11T01:47:42.249903000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (vmremap cd5a675*?) %
def hot_a(n):
    return sum(i * i for i in range(n))


def hot_b(n):
    return sum(i + i for i in range(n))


def worker():
    while True:
        hot_a(6_000_000)
        hot_b(6_000_000)


worker()
CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs
  • gh-152892
  • gh-153425
  • gh-153436

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Modules/_remote_debugging/binary_io.h and binary_io_writer.c, then inspect Lib/profiling/sampling/binary_collector.py and the linked PRs gh-152892, gh-153425, and gh-153436. Reproduce with the provided profiling command and verify that high-rate sampling no longer raises OverflowError or leaves the binary output corrupted.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
observability, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.