elastic / elastic/apm-server

PoC: store well defined metrics as times-series data streams

Open
#9,649 8 comments 0 reactions 0 assignees View on GitHub
poc
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

In recent versions, Elasticsearch has introduced time-series data streams ([TSDS](https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html)) -- a type of data stream that is well suited to storing (and querying) metrics. TSDS reduces disk space usage, and in the future it is expected to provide improved metric aggregations functionality. TSDS enables downsampling (rollup) of metrics, a feature that would enable our users to trade fidelity for cost, to maintain precision of metrics over a longer period for a reasonable cost.

Let's investigate changing the internal metrics data stream to use `index_mode: time_series`. Metrics will be identified and marked with the `time_series_metric` attribute. Metric dimensions (e.g. `service.name`) will be identified and marked with the `time_series_dimension` attribute.

We should investigate whether we can switch over to TSDS without affecting the UI, or if additional changes are required.

We should use Rally to identify any storage savings (or unexpected costs), ingest throughput degradation, and ideally query performance improvements.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.