Graylog2 / Graylog2/graylog2-server
Store gl2_receive_timestamp and gl2_processing_timestamp in indexed messages
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
In 3.1 we added the `gl2_receive_timestamp` and `gl2_processing_timestamp` fields to the `Message` class and used them for tracking processing status. (see https://github.com/Graylog2/graylog2-server/issues/5883 and https://github.com/Graylog2/graylog2-server/pull/6022)
We couldn't store the fields in the messages in Elasticsearch because our internal index template had a dynamic mapping that forced all `gl2_` prefixed fields to the `keyword` type. For timestamps we want the `date` type to be able to run date aggregations and queries on it.
In 3.1 we added both fields to the index mapping (https://github.com/Graylog2/graylog2-server/pull/6027) to make sure most users will have the field mappings in their active write indices when they update to 3.2. Users that skip 3.1 (not recommended) or having very long running write indices will not have the new mapping for their write indices. For those we need to run a migration to add the `gl2_receive_timestamp` and `gl2_processing_timestamp` fields to all current write indices.
This migration must run before any messages will be indexed. We can do that in the bootstrap/pre-flight check phase once we implemented https://github.com/Graylog2/graylog2-server/issues/5248.
Once that's done we can actually store both timestamps in the `Message` fields so they will be indexed.
Contributor guide
Assessment
This issue has not been assessed yet.