Graylog2 / Graylog2/graylog2-server
Track timestamp watermark of last received data per collector instance
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## What?
In the collector input(s), maintain a last received timestamp for each collector instance and regularly flush this information to the database.
There's no need to have this information in real time, so a reasonable flush delay is acceptable to reduce load and noise: the alerting on this information isn't going to be real-time either, and batched log ingest is also not streaming, so latency in the signal is the norm.
Tracking the receive timestamp is enough: we don't need the actual timestamp of the log records. We are only interested in whether we are receiving actual data.
## Why?
To show that a collector is not only connected via its supervisor but also sends log data, we need to capture that information.
Alternatively, we could search for log data across indices, but that adds load and is subject to indexing delays.
Contributor guide
Assessment
This issue has not been assessed yet.