elastic / elastic/logstash

Rate limit filter

Open
#4,726 11 comments 0 reactions 0 assignees View on GitHub
enhancement high hanging fruit
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Often times than not you don't really care if some hosts have thousands or millions of errors per second, the fact that they have more than 10 per second is enough to care about. Unfortunately, with logstash you cannot selectively drop messages when rate approaches levels when you don't care anymore.

I propose to add "ratelimit" filter with the following settings:
- `key` — key template for rate limit, could be something like `%{host}` or `%{host}/%{service}`.
- `interval` — time interval to limit messages in, could be 1 minute or 10 minutes for example.
- `limit` — max number of messages with given key per interval.

It can optionally add the number of dropped messages for key to the first message after `interval` reset.

While I have high doubts that logstash is really suitable for the cases when your load exceeds capacity of what jruby can comfortably do in terms of requests per second, I am also lazy and want to reuse existing stuff that I don't have to write and/or support.

The plugin itself should be pretty trivial to implement from what I can tell.

Let me know what you think.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.