Graylog2 / Graylog2/graylog2-server

`gl2_original_timestamp` causees data loss

Open
#23,025 8 comments 2 reactions 1 assignee Claimed by @kodjo-anipah View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

Graylog now detects timestamps in the future and replaces them with a sane value, which is nice. But it stores the old value in `gl2_original_timestamp` and defines that field as a "date" in the index mapping. This causes index failures if the value contains an invalid date string.

Practical example: Some Kubernetes logging solution generated timestamps that look like this: `556925-01-21 05:58:28.000`. Probably because someone mixed up seconds and microseconds for a timestamp given to an iso date function.

## Expected Behavior
Graylog should detect all invalid `timestamp` values (not only dates in the future but also parsing errors) and replace broken timestamps with a known-good fallback value (e.g. `gl2_receive_timestamp`). It should NOT copy invalid dates into internal date fields that will later cause index errors anyway.

## Current Behavior
If Graylog receives a message with an invalid timestamp (e.g. `556925-01-21 05:58:28.000`) than it copies that string to `gl2_original_timestamp` which then causes index errors and data loss.

## Possible Solution
Define `gl2_original_timestamp` as a keyword or string field (which cannot be done by users via Field Type Profiles because the internal mapping type is hardcoded). Or have a new field called `gl2_invalid_timestamp` that is a string value and used if the original timestamp fails to parse. Or configure the mapping of `gl2_original_timestamp` as [ignore-malformed](https://docs.opensearch.org/docs/latest/field-types/mapping-parameters/ignore-malformed/). Throwing such events away is not a reasonable default action to take.

* Graylog Version: 6.2.3+b5538f9
* OpenSearch Version: 2.15.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.