aio-libs / aio-libs/aiokafka

Add lightweight consumer metrics collector

Abierto
#1,177 2 comentarios 2 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
1.4k
Forks
269
Merge medio
1 d 1 h
PR fusionados (30 d)
6

Descripción

## Context

Follow-up to the producer metrics API from #1166 / #1171.

`AIOKafkaConsumer` has no equivalent of `metrics_collector`. Teams that need fetch latency, lag, throughput, or commit timing still have to wrap application code and cannot see fetcher / coordinator internals (#38, #562).

## Proposal

Add `ConsumerMetricsCollector` using the same pattern as `ProducerMetricsCollector`:

- sync callbacks on the hot path
- no aggregation / sampling / backend inside aiokafka
- keyword-only callback arguments
- no-op defaults so subclasses override only what they need
- `metrics_collector=` on `AIOKafkaConsumer`
- experimental until the shape stabilizes

## Initial callback set

| Callback | Intent |
|---|---|
| `on_fetch_completed` | fetch RTT + records/bytes returned (per topic-partition) |
| `on_fetch_failed` | fetch error after retries / terminal failure |
| `on_records_consumed` | records/bytes handed to the application via `getone` / `getmany` |
| `on_records_lag` | highwater − position after a fetch update |
| `on_commit_completed` | offset-commit latency |
| `on_commit_failed` | offset-commit failure |

`topic` and `partition` always passed where applicable; collectors decide on label cardinality. Durations in seconds.

## Out of scope for the first PR

- rebalance / heartbeat lifecycle metrics
- connection / broker-level metrics
- OpenTelemetry helper package

## Implementation notes

- Mirror producer conventions (`Null*` / default no-ops, typo warning for unknown `on_*`, exceptions logged and ignored).
- Likely hook points: `Fetcher` (fetch + lag), consumer poll path (records consumed), group coordinator (commits).
- Docs + example extension of the existing Prometheus example; `CHANGES.rst` entry.

## Questions

1. Is this initial callback set the right first cut?
2. Should lag be reported from the fetcher on every successful fetch, or only when the application consumes?

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.