GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver
[P1] event-exporter: bounded retries with backoff for Cloud Logging writes
- Dominant language
- Go
- Stars
- 409
- Forks
- 236
- Avg merge
- 2h 34m
- Merged PRs (30d)
- 9
Description
## Problem
`writer.Write` retries indefinitely with a fixed 10s delay. During API outages or throttling, this can stall throughput and cause backlog growth without a clear signal or drop policy.
## Proposed optimization
Introduce bounded retry with exponential backoff + jitter, and expose metrics for retry count and dropped batches. Optionally apply per-status handling (e.g., retry 429/5xx, fail fast on 400). This makes failure behavior predictable and prevents infinite backlog under sustained failure.
## Notes / references
- Cloud Logging WriteEntries API: https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/write
- Kubernetes client-go workqueue rate limiters (example for backoff): https://pkg.go.dev/k8s.io/client-go/util/workqueue
## Acceptance criteria
- Retries use exponential backoff with jitter and a max retry time or count.
- Metrics expose retry attempts and dropped batches.
- Behavior for 400 vs 429/5xx is explicit and documented.
Contributor guide
Assessment
This issue has not been assessed yet.