Graylog2 / Graylog2/graylog2-server

Store gl2_receive_timestamp and gl2_processing_timestamp in indexed messages

Open
#6,028 0 comments 1 reaction 0 assignees View on GitHub
elasticsearch feature processing triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

In 3.1 we added the `gl2_receive_timestamp` and `gl2_processing_timestamp` fields to the `Message` class and used them for tracking processing status. (see https://github.com/Graylog2/graylog2-server/issues/5883 and https://github.com/Graylog2/graylog2-server/pull/6022)

We couldn't store the fields in the messages in Elasticsearch because our internal index template had a dynamic mapping that forced all `gl2_` prefixed fields to the `keyword` type. For timestamps we want the `date` type to be able to run date aggregations and queries on it.

In 3.1 we added both fields to the index mapping (https://github.com/Graylog2/graylog2-server/pull/6027) to make sure most users will have the field mappings in their active write indices when they update to 3.2. Users that skip 3.1 (not recommended) or having very long running write indices will not have the new mapping for their write indices. For those we need to run a migration to add the `gl2_receive_timestamp` and `gl2_processing_timestamp` fields to all current write indices.

This migration must run before any messages will be indexed. We can do that in the bootstrap/pre-flight check phase once we implemented https://github.com/Graylog2/graylog2-server/issues/5248.

Once that's done we can actually store both timestamps in the `Message` fields so they will be indexed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.