Investigate potential memory leak in ratelimiter

Summary

The ratelimiter service's memory usage grows steadily over time (observed via Grafana dashboards) and pods eventually OOM after a few weeks of uptime, depending on traffic volume. There may be more than one leak involved — in the ratelimiter service code itself, in the olric library it embeds, or both.

Known leads

1. Olric eviction timer (unconfirmed theory): evictKeysAtBackground in olric runs an infinite loop calling time.After(100 * time.Millisecond) on each iteration. Timers created this way are not always released promptly and could accumulate over long uptimes. (not verified empirically)

2. Immortal (no-TTL) keys in olric's dmap store (measured, root cause identified): Captured in staging on ratelimiter 1.17.3 via the service's own :6061/debug/olric/keys debug endpoint, 60s after stopping a 110k req/min/pod ghz load run against a policy with a 1s TTL per key:

   Total Keys:          11522
   Keys With TTL:       0
   Keys WITHOUT TTL:    1776
   Scan Duration:       2139ms
   
   WARNING: Found 1776 immortal keys (no TTL set)
   

   These 1776 keys persisted with no TTL after load stopped, so they are not in-flight writes — they are stuck permanently.

   Root cause is in olric_store.go, Store.Get, which can write a key up to twice, and only one of the three write paths sets a TTL:

   • store.dmap.Put(ctx, fullKey, 1) in the ErrKeyNotFound branch — no TTL.
   • store.dmap.Put(ctx, fullKey, 1) in the "when ttl is 0, reset the value to 1" branch — no TTL.
   • store.dmap.Put(ctx, fullKey, value, olric.PX(rate.Period)) — the only write that sets a TTL, gated on value == 1 && rate.Period.Milliseconds() > 0.

   A key is created with no expiry and only later given one. Anything that stops the TTL-setting write from running or taking effect leaves that key immortal forever — nothing else in the code ever re-applies a TTL to a key that already lacks one. Candidate mechanisms (not yet confirmed which produces the observed keys):

   • rate.Period == 0 for some limiter tier, so the TTL-setting write never runs and the no-TTL write is final.
   • The TTL-setting write's error handler explicitly ignores ErrKeyNotFound ("// ignore key not found error"), so a failure there is silent and leaves the untimed key in place.
   • A crash or restart between the first (no-TTL) write and the second (TTL) write.

   Note the leak is not self-limiting: the "ttl is 0, reset value to 1" path rewrites an already-immortal key with another no-TTL write, so once a key goes immortal it stays immortal unless it happens to land on the TTL-setting write path.

   Why this matters beyond raw memory growth: MaxInuse is configured with LRUEviction. Immortal keys consume that memory budget permanently, which can push LRU into evicting live rate-limit counters — i.e., this isn't just a memory leak, it can also cause silent rate-limit resets under memory pressure.

   Suggested fix direction: always set the TTL on the initial Put rather than as a follow-up write. This also removes a second round trip per new key.

Ask

Looking for help narrowing down and fixing the leak(s) above. Contributions welcome — this is a good starting point for anyone wanting to get familiar with the ratelimiter service and its olric integration.