Rate limit filter
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
Often times than not you don't really care if some hosts have thousands or millions of errors per second, the fact that they have more than 10 per second is enough to care about. Unfortunately, with logstash you cannot selectively drop messages when rate approaches levels when you don't care anymore.
I propose to add "ratelimit" filter with the following settings:
- `key` — key template for rate limit, could be something like `%{host}` or `%{host}/%{service}`.
- `interval` — time interval to limit messages in, could be 1 minute or 10 minutes for example.
- `limit` — max number of messages with given key per interval.
It can optionally add the number of dropped messages for key to the first message after `interval` reset.
While I have high doubts that logstash is really suitable for the cases when your load exceeds capacity of what jruby can comfortably do in terms of requests per second, I am also lazy and want to reuse existing stuff that I don't have to write and/or support.
The plugin itself should be pretty trivial to implement from what I can tell.
Let me know what you think.
Contributor guide
Assessment
This issue has not been assessed yet.