python / python/cpython

Add Perfetto trace output to profiling.sampling

未关闭
#150,327 11 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

stdlib topic-profiling type-feature
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

Feature or enhancement

Proposal:

Feature

Add a Perfetto trace output backend to profiling.sampling (target: 3.16), alongside the existing other output formats. This would let users open Python sampling profiles directly in the Perfetto UI and analyse them with PerfettoSQL in the trace processor.

Opening this for design sanity-check before sending a PR, as suggested by @pablogsal in offline discussion. The most load-bearing question (how to emit protobuf without a runtime dependency) is covered below.

Motivation

  • Perfetto is a trace viewer oriented systems-perf work, and being able to view Python samples there means a Python profile can sit on the same timeline as scheduler events, native CPU samples, and application instrumentation rather than living in a separate tool.
  • Once trace processor can read the output, users get SQL-driven analysis of Python samples (top-N functions, per-thread breakdowns, custom aggregations).

Proposed design

Output format

Emit a Perfetto trace (perfetto.protos.Trace) containing:

  • A ProcessDescriptor / ThreadDescriptor per observed process/thread.
  • InternedData carrying Frame, Mapping, and Callstack entries
  • One TracePacket per sample containing a StackSample message that references the interned callstack.

StackSample is part of a new set of public profiling protos I'm landing in Perfetto specifically so that producers like this one have a stable, transport-neutral surface to target (rather than reusing PerfSample, which is shaped by perf_event_open and leaks producer diagnostics into the data). The RFC is at https://github.com/google/perfetto/discussions/6027.

Concretely, each Python sample maps to roughly:

TracePacket {
  timestamp: <ns>
  trusted_packet_sequence_id: <seq>
  interned_data { ... }  // first packet only, or as new frames appear
  stack_sample {
    task_context_iid: <thread>
    execution_context_iid: <cpu/mode>     // optional
    callstack_iid: <interned callstack>
    primary_descriptor_iid: <"wall_time_ns" counter>
    primary_weight: <ns since last sample>
  }
}
Proto serialisation without a runtime dependency

The stdlib can't depend on protobuf, so I'd hand-roll the wire format for the specific message types we emit. This is tractable because:

  • Proto wire format is tiny. It's varints + length-delimited + fixed32/64 + a tag byte per field. The whole encoder for the messages we need is on the order of a couple hundred lines of Python.
  • We only encode, never decode. Decoding protobuf is signifcantly harder than encode.
  • The set of message types is small and stable. I've specifically designated in the RFC upstream that these protos I'm adding are going to be "eternally stabe" protos which we won't change the wire format or semantics in a non-backwards compatible way.
  • Precedent. Perfetto itself ships an ad-hoc proto encoder/decoder (protozero) in C++ for similar reasons.

Sketch of the shape:

# Hand-rolled, no runtime deps.
def _varint(buf, n): ...
def _tag(buf, field_no, wire_type): ...
def _string(buf, field_no, s): ...
def _message(buf, field_no, payload): ...

def encode_stack_sample(buf, sample):
    _tag(buf, 1, WIRE_VARINT); _varint(buf, sample.task_context_iid)
    _tag(buf, 6, WIRE_VARINT); _varint(buf, sample.callstack_iid)
    # ...

Field numbers and wire types come straight from the .proto definitions in the Perfetto RFC. Any wire type constants would be inlined as Python integers.

CLI surface

A new --format perfetto CLI flag to the profiling.sampling module in all the same places --gecko is allowed today.

Target

Python 3.16.

Prerequisites

  • Perfetto RFC-0027: public stack-sampling and heap-profiling protos. Required before this lands so we're targeting the stable protos.

cc @pablogsal

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs
  • gh-154541

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

从 profiling.sampling CLI 开始,沿着现有的 --gecko 输出路径进行跟踪,然后阅读链接的 Perfetto RFC,了解 StackSample 及相关 proto 定义。当 --format perfetto 输出的 trace 能够在 Perfetto UI 中打开,并支持所描述的 Python sample 和 thread 数据时,这项工作就完成了。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
cli, performance
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。