GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver

[P1] event-exporter: add bounded queue + worker pool for sink batching

Open
#1,032 0 comments 0 reactions 0 assignees View on GitHub
area/event-exporter enhancement go kind/feature priority/important-soon
Dominant language
Go
Stars
409
Forks
236
Avg merge
2h 34m
Merged PRs (30d)
9

Description

## Problem
The sink enqueues individual log entries into a single buffered channel and spawns a goroutine per flush. This makes throughput sensitive to burst patterns and offers limited visibility into queue depth or backpressure.

## Proposed optimization
Introduce a bounded work queue + worker pool that batches by size/time with explicit queue depth metrics. This provides stable throughput under load and makes backpressure behavior observable.

## Notes / references
- client-go workqueue (rate-limiting + metrics): https://pkg.go.dev/k8s.io/client-go/util/workqueue
- Prometheus metrics for queue depth/latency can mirror workqueue defaults.

## Acceptance criteria
- Sink uses a bounded queue with worker pool and batcher.
- Metrics expose queue depth and batch flush latency.
- Throughput remains stable under bursty event streams.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.