Graylog2 / Graylog2/graylog2-server
Add configurable options for pipeline and pipeline rule timeouts
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## What?
Per a discussion on the forum about Graylog's output dropping to 0 due to a potentially stuck pipeline: it would be brilliant if we can configure some sort of high water mark for things like time spent on any single rule, as in, if a rule takes more than 500ms to complete, stop the pipeline, and store the message with a gl2_rule_timeout field or something indicating which rule and how much time it took and so on. IMO it’s better to have semi-processed logs stored than to bring things to a grinding halt
Ideally a similar option for an entire pipeline, so the ability to say if a message has been in a particular pipeline for more than X (time units), to also time it out and do the thing mentioned above.
I envision the pipeline one being an option that can be set on the pipeline's configuration, perhaps also the per-rule timeout should go there unless it's something that could be configured per rule. The "sane" defaults would be to set them to 0 to indicate unlimited time, as in, the current behaviour.
## Why?
Because any time a pipeline is stuck, the journal fills up, and when you're pushing anywhere from 10k to 20k msg/sec this can get out of control in a real big hurry. We have had to deal with this a few times, and trying to figure out which rule is potentially causing issues is not exactly easy. The metrics give a little bit of a clue, but since it only shows the internal ID for a rule and not it's name, and only throughput counts, it's not easy.
I'd rather be able to say that if a rule doesn't complete within X time, to just consider the pipeline as failed, mark the message, and store it so external tooling can be brought in to do the whole alert song and dance; at the very least logs still get stored, and if need be can be reprocessed manually at a later date.
Contributor guide
Assessment
This issue has not been assessed yet.