elastic / elastic/package-spec

[Change Proposal] Support look back time for TSDB indexes

Open
#1,179 0 comments 0 reactions 0 assignees View on GitHub
discuss
Dominant language
Go
Stars
20
Forks
93
Avg merge
15h 10m
Merged PRs (30d)
12

Description

Time series data streams have a writable window of 2 hours in the past. This means that documents whose `@timestamp` is older than these 2 hours are rejected.

There are integrations use cases where data is collected in longer periods, for example metrics that are published only daily. In these cases the 2 hours window is too small.

Elasticsearch supports [`index.look_back_time`](https://www.elastic.co/docs/reference/elasticsearch/index-settings/time-series) to change this 2 hours default.

Allow to use `index.look_back_time` in the index settings. This should be only allowed when time series is enabled.
For completitude, allow to use also `index.look_ahead_time`, that increases the window in the future.

As a workaround users can already override this value using `@custom` index templates, but it should work out of the box for integrations that are known to collect data older than 2 hours.

Feature requested in https://github.com/elastic/integrations/pull/19339#issuecomment-4613358008.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.