PoC: store well defined metrics as times-series data streams
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 543
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 109
Description
In recent versions, Elasticsearch has introduced time-series data streams ([TSDS](https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html)) -- a type of data stream that is well suited to storing (and querying) metrics. TSDS reduces disk space usage, and in the future it is expected to provide improved metric aggregations functionality. TSDS enables downsampling (rollup) of metrics, a feature that would enable our users to trade fidelity for cost, to maintain precision of metrics over a longer period for a reasonable cost.
Let's investigate changing the internal metrics data stream to use `index_mode: time_series`. Metrics will be identified and marked with the `time_series_metric` attribute. Metric dimensions (e.g. `service.name`) will be identified and marked with the `time_series_dimension` attribute.
We should investigate whether we can switch over to TSDS without affecting the UI, or if additional changes are required.
We should use Rally to identify any storage savings (or unexpected costs), ingest throughput degradation, and ideally query performance improvements.
Contributor guide
Assessment
This issue has not been assessed yet.