Graylog2 / Graylog2/graylog2-server
Alert Server Changes - Reliable Event Indexing
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
The current "persist-to-streams" event action writes the generated events directly into Elasticsearch. There is no error handling and retries to avoid losing events when something goes wrong. (e.g. Elasticsearch is down)
### There are a few options to do this
* Using an invisible, internal input that ingests the created events
* Would buffer the events in the journal and use the regular indexing methods (including the flawed error handling for rejected bulk index requests)
* Users could theoretically configure regular message processing components to enrich events
* Regular stream handling, extractors and pipeline systems would need special handling for events, though. Because the events schema is very different from messages and we cannot add/modify arbitrary fields
* Using a separate journal to buffer the created events
* Requires some custom code for buffering created events in a journal and custom error handling in case of indexing errors
* Recent forwarder changes made the Kafka journal reusable so it's pretty easy to setup a custom journal and journal handler.
* Clear separation between messages and events
* Possible to implement event specific error handling
Contributor guide
Assessment
This issue has not been assessed yet.