influxdata / influxdata/influxdb

Last 10 rows on big measurement

Open
#21,302 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

I've an issue since the migration to influxdb 2. We have ameasurement with thousands of data by day and I need to know the last 10 rows in a month.
So previously, we have the following query : select * from payload where time > now() - 4w and time < now() order by desc limit 10

Now we translated it with :
from(bucket: "mybucket")
|> range(start: -4w, stop: now())
|> filter(fn: (r) => r._measurement == "payload")
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|> group()
|> sort(columns: ["_time"], desc: true)
|> limit(n: 10)

But the time is very long !
It seems to be the "sort" function is very expensive.

Have you an idea to get my results faster ?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Flux query over a large measurement, then examine the range, pivot, group, sort, and limit stages to identify where execution time is spent. Compare the query’s behavior with the reported legacy query and measure any candidate change; done means the last 10 rows for the requested period are returned with a demonstrated performance improvement.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.