envoyproxy / envoyproxy/ratelimit
Memory leak when RateLimitOverride is set with uniq ip on every call to ShouldRateLimit method
- Dominant language
- Go
- Stars
- 2.7k
- Forks
- 535
- PR merge metrics
- No merged PRs in 30d
Description
In our service we use `envoyproxy/ratelimit` as sidecar and are making grpc calls to `ShouldRateLimit` method explicitly without envoy proxy.
Simplified config looks like
```
---
domain: app
descriptors:
- key: ip
rate_limit:
unit: minute
requests_per_unit: 1000
```
In our request we specify `Limit` (limit override) - https://github.com/envoyproxy/java-control-plane/blob/main/api/src/main/proto/envoy/extensions/common/ratelimit/v3/ratelimit.proto#L93 on every request which lead to this codepath getting executed - https://github.com/envoyproxy/ratelimit/blob/main/src/config/config_impl.go#L252-L261
In that if statement when unique IP is getting used on every request we are getting unique `rateLimitKey` and also as a result are getting new set of stats Counters `this.statsManager.NewStats(rateLimitKey),`. So number of stats Counters and keys always grows over time.
- Stats should not be collected at all if `USE_STATSD` is set to `false`.
- Stats should expire and stats cache should get cleaned up over time
**UPDATE**:
our simplified ratelimit request to looks like this:
```
{
"domain": "app",
"descriptors": [
{
"entries": {
"key": "our_entry_key",
"value": "customer1234_1.2.3.4"
},
"limit": {
"requests_per_unit": 1000,
"unit": "MINUTE"
}
}
],
"hits_addend": 1
}
```
^ note 2 things
- we have `limit` defined in each request and is different based on customer. In this case customer is `customer1234`
- key is built by `_`
Because `limit` is defined this code path is getting executed
https://github.com/envoyproxy/ratelimit/blob/main/src/config/config_impl.go#L252-L261
which executes `this.statsManager.NewStats(rateLimitKey)` which will build whole bunch of new stats counters based on `key` (`customer1234_1.2.3.4`)
Contributor guide
Assessment
This issue has not been assessed yet.