influxdata / influxdata/influxdb

Automatic downsampling and querying (rrd-like)

Open
#23,108 5 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

The downsampling and retention in Influx is very administration intensive without generic functions. E.g., it is hard or impossible to do a post-downsampling of data of several years to 30 minute averages, because there are only downsampling mechanisms that run from 'now' minus x minutes.

The second issue is, that downsampling creates new measurement names and the queries need to explicitely specify the destination database to get high-accurate or downsampled data. There is no auto-select.

Coming from rrdtool, I'd like to request the following features:

- Generic downsampling query that can downsample all data in all measurements of a database without need of explicitely name the measurements.
- Measurement names in the downsampled databases are equal to the original database with "mean" values.
- Downsampling can take place also afterwards. When data of 2 years are imported, and retention policy says 30min average, this also works for the 2 years of data, not only for the last 30 minutes.
- The Retention and downsampling process creates databases that are connected to the default database. Influx should be aware of what is the live database, and what downsampling granularity is located in what "connected" databases.
- On queries, there is no need to use the specific downsampled database name. Instead, Influx decides by the query time range if it could use the default database, or need to query the connected downsampled databases. Influx knows the accuracy of every connected database and automatically selects the database that can fulfill the request with best accuracy.

This is in summary what Tobias Oetiker's rrdtool is doing, that would easify working with retention also in Influx.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points to begin with. Start by reviewing InfluxDB's retention and downsampling implementation, then assess the requested generic historical downsampling and automatic database selection. Done would require an agreed design and implementation covering the listed retention, naming, connection, and query-selection behaviors.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.