GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver

[P1] event-exporter: bounded retries with backoff for Cloud Logging writes

Open
#1,031 0 comments 0 reactions 0 assignees View on GitHub
area/event-exporter enhancement go kind/feature priority/important-soon
Dominant language
Go
Stars
409
Forks
236
Avg merge
2h 34m
Merged PRs (30d)
9

Description

## Problem
`writer.Write` retries indefinitely with a fixed 10s delay. During API outages or throttling, this can stall throughput and cause backlog growth without a clear signal or drop policy.

## Proposed optimization
Introduce bounded retry with exponential backoff + jitter, and expose metrics for retry count and dropped batches. Optionally apply per-status handling (e.g., retry 429/5xx, fail fast on 400). This makes failure behavior predictable and prevents infinite backlog under sustained failure.

## Notes / references
- Cloud Logging WriteEntries API: https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/write
- Kubernetes client-go workqueue rate limiters (example for backoff): https://pkg.go.dev/k8s.io/client-go/util/workqueue

## Acceptance criteria
- Retries use exponential backoff with jitter and a max retry time or count.
- Metrics expose retry attempts and dropped batches.
- Behavior for 400 vs 429/5xx is explicit and documented.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.