Free-threading: add stop-the-world duration and contention counts to pystats
還沒有人認領這個 Issue。
- 主要語言
- Python
- 星號
- 77.2k
- 分支
- 36k
- 平均合併
- 1 天 9 小時
- 30 天內合併 PR
- 558
描述
Feature or enhancement
Proposal:
gh-131253 added the first free-threading counters to pystats (mutex_sleeps, world_stops, qsbr_polls) and was recently closed as complete. After spending some time measuring the free-threaded build with them, I think two small additions would make them considerably more useful. The numbers below are from current main on a 24-core Linux machine, using Tools/ftscalingbench plus uprobe tracing on _PyMutex_LockTimed to cross-check what the counters report.
Stop-the-world duration. world_stops counts pauses but says nothing about their cost. Two ftscalingbench runs I traced both did exactly 277 stops, but total pause time was 0.45 ms in one (pymethod) and 1.53 ms in the other (staticmethod_call) - a 3.4x difference the counter cannot distinguish. The original issue already floated "possibly total duration"; it just never got implemented. Two clock reads in stop_the_world() would cover it. A max would be nice as well, since the distribution has a long tail - most pauses I saw were 0.5-2 us, with outliers around 16 us.
Contention events, not just sleeps. mutex_sleeps counts a thread actually parking, but _PyMutex_LockTimed spins first, so any contention that resolves by spinning is invisible. On a deliberately contended workload (8 threads appending to one shared list) I measured 196 entries into _PyMutex_LockTimed against a mutex_sleeps of 14, so the one contention counter we have misses roughly 93% of contention. An entry counter in _PyMutex_LockTimed would close that gap, and since it sits on the slow path already the cost is one relaxed increment.
Two more observations from the same experiments, recorded here because they shaped what I'm not proposing:
The scaling losses in ftscalingbench are not lock waits. Three workloads losing 35-50% of ideal scaling at 8 threads each showed only about 0.5 ms of genuine lock waiting, nearly all of it the import machinery RLock during thread startup plus interpreter teardown - none of it in the benchmark loops themselves. So I don't think more lock counters beyond the above are worth adding; whatever those benchmarks are losing, it happens below the mutex layer (my guess is refcounting cache traffic), and hardware-level tools are the right instrument there.
Benchmarking the free-threaded build also has a warmup trap worth knowing about: bytecode is thread-local, so specialization warmup on one thread does nothing for workers spawned later. I initially measured cold-bytecode churn without realizing it - FOR_ITER re-specializing hundreds of times per run, disappearing entirely when I reversed the run order. Anything that benchmarks with freshly spawned threads is exposed to this. Happy to write it up for the free-threading HOWTO if there's interest.
I can prepare PRs for both counters. This deliberately doesn't touch ring-buffer stats support for the FT build, which came up at the end of gh-131253.
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
No response
Linked PRs
- gh-155069
貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
研究方向
從 stop_the_world() 和 _PyMutex_LockTimed 周邊的 pystats free-threading 實作開始,接著使用 Tools/ftscalingbench 重現所述的測量結果。完成的標準是 pystats 回報 stop-the-world 的總持續時間和 mutex 競爭進入次數;請先檢視連結的 PR gh-155069,因為相關工作已在進行中。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- python
- 領域
- performance
- Issue 類型
- 功能
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 停滯
- 描述清晰度
- 基本清楚
- 新手友好度
- 30/100