Graylog2 / Graylog2/graylog2-server

Add fault tolerance layer for components connecting to external systems

Open
#6,536 0 comments 0 reactions 0 assignees View on GitHub
feature needs-discussion triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior
When connecting to external systems during message processing, those systems can become slow or stop responding altogether.
Ideally the user or admin would be able to specify what the system is doing when this happens:
* stop processing until the service is back up
* continue with a sane default
* skip this part of processing
* reprocess once the service is back up

## Current Behavior
Lookup tables are especially prone to this problem, and right now many of them ship with long defaults which can completely stall processing when disaster strikes.
E.g. if OTX is down any reasonably busy Graylog server will grind to a halt and eventually lose data.
The user has no option to say whether or not that enrichment step is critical or not, and worse, it might affect unrelated streams of data.

## Possible Solution
* split message processing by stream?
* add something like https://github.com/Netflix/Hystrix to have one layer to deal with unavailable resources?
* ability to pause certain processing stages?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.