Graylog2 / Graylog2/graylog2-server

Add configurable options for pipeline and pipeline rule timeouts

Open
#13,583 1 comment 5 reactions 0 assignees View on GitHub
feature needs-discussion triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## What?
Per a discussion on the forum about Graylog's output dropping to 0 due to a potentially stuck pipeline: it would be brilliant if we can configure some sort of high water mark for things like time spent on any single rule, as in, if a rule takes more than 500ms to complete, stop the pipeline, and store the message with a gl2_rule_timeout field or something indicating which rule and how much time it took and so on. IMO it’s better to have semi-processed logs stored than to bring things to a grinding halt

Ideally a similar option for an entire pipeline, so the ability to say if a message has been in a particular pipeline for more than X (time units), to also time it out and do the thing mentioned above.

I envision the pipeline one being an option that can be set on the pipeline's configuration, perhaps also the per-rule timeout should go there unless it's something that could be configured per rule. The "sane" defaults would be to set them to 0 to indicate unlimited time, as in, the current behaviour.

## Why?
Because any time a pipeline is stuck, the journal fills up, and when you're pushing anywhere from 10k to 20k msg/sec this can get out of control in a real big hurry. We have had to deal with this a few times, and trying to figure out which rule is potentially causing issues is not exactly easy. The metrics give a little bit of a clue, but since it only shows the internal ID for a rule and not it's name, and only throughput counts, it's not easy.

I'd rather be able to say that if a rule doesn't complete within X time, to just consider the pipeline as failed, mark the message, and store it so external tooling can be brought in to do the whole alert song and dance; at the very least logs still get stored, and if need be can be reprocessed manually at a later date.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.