Rate-limit counters reset when ratelimiter pods are replaced
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Rate-limit counters are held only in the ratelimiter's Olric cluster memory, and they are lost when the pods holding them go away. A caller then regains its full budget in the middle of a window, with nothing logged and no metric to distinguish it from a new window.
Reproduced on a self-hosted install with three ratelimiter pods and a function limited to 10 per hour:
- Spend 4 of the budget, replace all three pods one at a time, then invoke again: all 10 further calls are admitted, so 14 are admitted against a limit of 10.
- Same sequence with no pod replacement: exactly 10 admitted, then denied. Enforcement itself is correct.
- Replacing only one or two of the three pods keeps the counter, because a pod that still holds a copy survives.
Cause: each counter lives on the member that owns its partition, and the Olric replica count is left at the default of 1, so there is no second copy. Olric also only seeds a member on write. Its balancer moves a partition when the current holder stops owning it, and nothing back-fills a backup owner that joined while a counter sat idle, so a freshly started pod holds nothing for existing counters. Interleaving a single request between pod replacements is enough to keep the counter, which confirms writes are the only path that seeds a new member.
Impact: a rolling upgrade of the ratelimiter replaces every pod in turn, so every counter is lost and enforcement is effectively off for the duration of the rollout. Any pod churn (eviction, node drain, OOM kill, scale-down) has the same effect on the counters that pod held.
Suggested direction: give each counter a backup copy, hand a terminating pod's counters to the current owners before it exits, and surge the replacement in first so there is a live member to receive them.
Note that this still would not make counters durable: losing every member at once has nothing to recover from, since nothing is persisted.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the ratelimiter's Olric cluster configuration and the pod termination and rollout entry points. Reproduce the three-pod replacement sequence, then verify that counters retain a backup or are handed to a live member during replacement, while confirming that losing every member remains non-durable as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100