Graylog2 / Graylog2/graylog2-server
Decouple Outputs from Indexes
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
Right now, Outputs are tied to Streams, and Indexes are also tied to streams. This means that Outputs are indirectly tied to Indexes. Indexes are used to manage, among other things, retention and rotation strategies; Outputs are of course used to manage forwarding information to other systems.
These two considerations are orthogonal to each other, but currently you have to be careful when setting up your Outputs and Streams to make sure you don't accidentally put messages into the wrong index and thus give them the wrong retention behavior (or, conversely, if you start routing messages to a different index so they'll get different retention behavior, you may also stop them from getting sent to the right outputs).
## Example situation
Consider messages which have two fields, `application` and `severity`. We have two applications, 'foo' and 'bar'. We set up two indexes, because messages from 'foo' need to be retained for a month while messages from 'bar' need to be retained for 6 months.
We also want to set up an output that forwards all messages with a severity of 'error' or higher to another system, regardless of which application the message is from.
## Expected Behavior
We set up two indexes, and one output, some number of streams less than `indexes * (outputs + 1)` in order to facilitate it all. Every message gets indexed exactly once, in the index that has the correct retention policy.
## Current Behavior
We have to set up four streams, `foo`, `foo-error`, `bar`, and `bar-error`. This gets much worse as you add more outputs and indexes.
If you have less than four streams, you end up with messages going into the wrong index because that was required by the output they were being routed to.
## Possible Solution
Decouple Indexes from Streams, and add `Index Rules` that are similar to Stream Rules but applied directly to the Index, along with Pipeline functions that route messages to an Index rather than to a stream.
OR
Decouple Outputs from Streams, and add `Output Rules` that function similar to Stream Rules but are applies directly to the Output, along with Pipeline functions that route messages to an Output rather than a Stream
OR
Allow Streams to exist with no associated Index; routing a message to such a stream would process outputs (and any other Stream-specific features) but would not add the message to any additional indexes that it wasn't already being routed into by other means.
## Context
We want to start putting messages from certain applications into a different index so that they can be retained for longer, but it means redoing all of our Streams to make sure the messages don't end up indexed in multiple different indexes.
## Your Environment
* Graylog Version: 3.1.2
* Elasticsearch Version: 6.5.2
* MongoDB Version: 3.0.15
* Operating System: Centos 7.4
Contributor guide
Assessment
This issue has not been assessed yet.