Improve storage efficiency for Metrics (TSDB): New storage Codec (ES95)
- Dominant language
- No language data
- Stars
- 6
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## Value Prop
Observability teams rely on metrics to monitor system health and diagnose production incidents. The high volume of this data forces a trade-off between storage costs, data completeness, and the speed of analysis during an outage. This means teams often have to limit monitoring to critical services or reduce data retention periods, creating blind spots that can prolong investigations.
With the storage efficiency updates for TSDB released in Elasticsearch versions 9.1-9.4, we have transformed Elasticsearch to a [highly competitive platform for storing time series and metrics](https://www.elastic.co/search-labs/blog/elasticsearch-columnar-metrics-engine-30x-faster-prometheus). The optimizations introduced in Elasticsearch 9.4 have brought the storage required per sample to 3.74 bytes per sample (in our internal benchmarks) down from 10.8 bytes per sample in Elasticsearch 9.3. That’s a 2.9x improvement in storage efficiency in a single release, with an overall improvement of 6.6x since we started this effort.
The last remaining optimization is to further improve the codec we use internally, which defines how we store and compress the various fields in our columnar store.
## Expected Outcome
Our existing codec provides already up to 6x better storage efficiency compared to Lucene's codec, but we know that there are further optimizations that we can introduce that can allow us to store numeric values more efficiently.
According to early internal benchmarks, we expect the new ES95 codec to offer:
- Better compression for double/float fields
- Utilize additional compression steps, like delta of deltas, or Run Length Encoding (RLE) for numeric fields to increase storage efficiency (especially for timestamps)
- Reduce the overall storage required per sample in our internal metrics benchmarks to around ~3.4 bytes per sample from 3.74 bytes per sample in 9.4, a **~10%** further reduction in storage used.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.