NVIDIA / NVIDIA/nvcf

Rate-limit counters reset when ratelimiter pods are replaced

Open
#975 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Rate-limit counters are held only in the ratelimiter's Olric cluster memory, and they are lost when the pods holding them go away. A caller then regains its full budget in the middle of a window, with nothing logged and no metric to distinguish it from a new window.

Reproduced on a self-hosted install with three ratelimiter pods and a function limited to 10 per hour:

  • Spend 4 of the budget, replace all three pods one at a time, then invoke again: all 10 further calls are admitted, so 14 are admitted against a limit of 10.
  • Same sequence with no pod replacement: exactly 10 admitted, then denied. Enforcement itself is correct.
  • Replacing only one or two of the three pods keeps the counter, because a pod that still holds a copy survives.

Cause: each counter lives on the member that owns its partition, and the Olric replica count is left at the default of 1, so there is no second copy. Olric also only seeds a member on write. Its balancer moves a partition when the current holder stops owning it, and nothing back-fills a backup owner that joined while a counter sat idle, so a freshly started pod holds nothing for existing counters. Interleaving a single request between pod replacements is enough to keep the counter, which confirms writes are the only path that seeds a new member.

Impact: a rolling upgrade of the ratelimiter replaces every pod in turn, so every counter is lost and enforcement is effectively off for the duration of the rollout. Any pod churn (eviction, node drain, OOM kill, scale-down) has the same effect on the counters that pod held.

Suggested direction: give each counter a backup copy, hand a terminating pod's counters to the current owners before it exits, and surge the replacement in first so there is a live member to receive them.

Note that this still would not make counters durable: losing every member at once has nothing to recover from, since nothing is persisted.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the ratelimiter's Olric cluster configuration and the pod termination and rollout entry points. Reproduce the three-pod replacement sequence, then verify that counters retain a backup or are handed to a live member during replacement, while confirming that losing every member remains non-durable as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.