influxdata / influxdata/influxdb
Influxdb - Grafana stack is not scalable due to unsupported time roll up
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Influxdb has the nice feature to roll up time serie data into larger time buckets via continous queries. This can tremendously speed up a query that needs to display two years of data that was originally sampled at 1Hz.
Grafana will however creates a select query on the original time serie and request a dynamic groupby time based on how far it is zoomed out. Influx will thus crunch all data points in that two year interval. There is no way in Grafana to specify to switch to the time rolled up continous query based on the groupby interval. This could potentially fixed in Grafana, however...
Grafana points out that Graphite automatically rollup its measurements as part of there measurement definition. An queries will thus automatically pick the best internally precalculated time roll up table to execute the query. Graphite therefore always answers within about 100ms. My experience with Influx is that it can take 30sec easily to return the result of a very zoomed out time serie. See https://github.com/grafana/grafana/issues/420 for more details.
This is basically a very nasty interface problem between Grafana and Influxdb and I see no urgency on either side to solve it while it is the number one performance bottleneck in this popular stack as far as I'm concerned.
The question:
Can the influx query engine use the knowledge its continous queries definitions to optimise the query engine path to use the pre calculated continous query data when a time groupby is requested on the original serie?
Contributor guide
Research direction
No files, tests, or entry points are named. Start by clarifying how continuous-query definitions relate to queries with time-based grouping, then document the expected selection behavior and a measurable performance target. Done means an agreed implementation path for using precomputed rollups without changing query results.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100