python / python/cpython

Add streaming to `profiling.sampling`

Abierto
#145,464 7 comentarios 4 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

stdlib topic-profiling type-feature
Lenguaje dominante
Python
Estrellas
77.2k
Forks
35.9k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

Feature or enhancement

Proposal:

Right now, profiling.sampling has roughly two modes: live with the TUI, and snapshot-at-the-end (except binary? but see the note below.) There's nothing that streams the data continously as it comes. This would be ideal for long-running headless profiling.

This one is less defined than #145411, so there are more open questions:

  • Should it stream raw or agreggate data? What should be the window?
  • What should be the format? Unfortunately, from what I checked the current binary format is not really well-suited for streaming, as it saves the dictionaries only on finalize.
  • What should be the transport layers for streaming?
  • What should be the types of messages?
  • What should be the configuration flags?
  • How the backpressure should be handled? Should it drop the oldest? All the oldest?

My hunch is:

  • support both raw and aggregate and assume that aggregate is just a different message type.
  • start with something simple as JSONL
  • I have mixed feelings about the transport layer. The stdout sounds great on paper but it's a mixed-use channel right now and there's a question around blocking by slower consumers. Maybe for debugging? Unix socket is good but not perfectly portable
  • as for backpressure, I would just drop the oldest by default, but only the raw, agg and heartbeat messages.

For starters:

  • --stream unix:/tmp/tachyon-stream.sock or --stream file:/tmp/tachyon-stream.sock (later: --stream tcp:127.0.0.1:1234)
  • --stream-types meta|str_def|frame_def|raw|agg:5s|loss|heartbeat|end|error (comma separated; note the agg:5s here; we it could be extended to heartbeat:1s etc. in the future)
  • --stream-format jsonl

Then, in the next phases we could think of --stream-drop-policy etc.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Línea de trabajo

Empieza leyendo la implementación de profiling.sampling y la discusión relacionada en el issue #145411. Revisa los tipos de stream propuestos, el formato JSONL, los transports y las cuestiones de backpressure antes de acotar el diseño. Se considera terminado cuando haya un alcance de streaming acordado y una configuración que admita profiling headless de larga duración.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
performance
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Tranquilo
Claridad
Necesita aclaración
Aptitud para principiantes
25/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.