influxdata / influxdata/influxdb

Influxdb - Grafana stack is not scalable due to unsupported time roll up

Open
#4,605 14 comments 1 reaction 0 assignees View on GitHub
1.x area/continuous queries area/queries kind/feature-request pm/review
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Influxdb has the nice feature to roll up time serie data into larger time buckets via continous queries. This can tremendously speed up a query that needs to display two years of data that was originally sampled at 1Hz.

Grafana will however creates a select query on the original time serie and request a dynamic groupby time based on how far it is zoomed out. Influx will thus crunch all data points in that two year interval. There is no way in Grafana to specify to switch to the time rolled up continous query based on the groupby interval. This could potentially fixed in Grafana, however...

Grafana points out that Graphite automatically rollup its measurements as part of there measurement definition. An queries will thus automatically pick the best internally precalculated time roll up table to execute the query. Graphite therefore always answers within about 100ms. My experience with Influx is that it can take 30sec easily to return the result of a very zoomed out time serie. See https://github.com/grafana/grafana/issues/420 for more details.

This is basically a very nasty interface problem between Grafana and Influxdb and I see no urgency on either side to solve it while it is the number one performance bottleneck in this popular stack as far as I'm concerned.

The question:
Can the influx query engine use the knowledge its continous queries definitions to optimise the query engine path to use the pre calculated continous query data when a time groupby is requested on the original serie?

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by clarifying how continuous-query definitions relate to queries with time-based grouping, then document the expected selection behavior and a measurable performance target. Done means an agreed implementation path for using precomputed rollups without changing query results.

Written by the indexing model from the issue text.

Assessment

Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.