GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver
[P1] event-exporter: add bounded queue + worker pool for sink batching
- Dominant language
- Go
- Stars
- 409
- Forks
- 236
- Avg merge
- 2h 34m
- Merged PRs (30d)
- 9
Description
## Problem
The sink enqueues individual log entries into a single buffered channel and spawns a goroutine per flush. This makes throughput sensitive to burst patterns and offers limited visibility into queue depth or backpressure.
## Proposed optimization
Introduce a bounded work queue + worker pool that batches by size/time with explicit queue depth metrics. This provides stable throughput under load and makes backpressure behavior observable.
## Notes / references
- client-go workqueue (rate-limiting + metrics): https://pkg.go.dev/k8s.io/client-go/util/workqueue
- Prometheus metrics for queue depth/latency can mirror workqueue defaults.
## Acceptance criteria
- Sink uses a bounded queue with worker pool and batcher.
- Metrics expose queue depth and batch flush latency.
- Throughput remains stable under bursty event streams.
Contributor guide
Assessment
This issue has not been assessed yet.