influxdata / influxdata/influxdb
Monolith OSS - Limit query consumes high CPU usage for look backs 12+hours
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
### Problem 1
The SQL limit query consumes very high CPU utilization even for a single series. On a `m6dg.large` machine which is of 2 cores and 8 GiB RAM, we observe that the query uses more than 75% and in many cases nearly 95%.
### Problem 2
As the LIMIT increases, drastic increase in CPU usage is observed. LIMIT 1 -> LIMIT 5.
Note: Happens only in OSS. This has an effect on the latency as well and these experiments are without concurrency.
### Configuration
The problem was found with the below basic setup,
- 2 cores, 8 GiB RAM
- Light Ingest, about 17 rows/second
- Executed as a stand-alone query (tried as part of a suite as well, same concern)
- Query look back used was 12+ hours (approx. 734400 rows scanned)
### Queries
> SELECT * FROM senec_data WHERE device_id = '10' and time > (now() - interval'{{.QueryRange}} minute')::timestamp ORDER BY time DESC LIMIT 1;
>
> SELECT * FROM senec_data where device_id = '10' and time > (now() - interval'{{.QueryRange}} minute')::timestamp order by time desc limit 5;
### Metrics/Evidence
Q5 query - LIMIT 5 reaching 93.5%
F1 query - LIMIT 1 reaching 74.4%
Latency over 100 runs
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files or tests are named. Start by reproducing the two SQL queries with the reported 12+ hour lookback and approximately 734,400 scanned rows, comparing LIMIT 1 and LIMIT 5 on the described setup. Done means identifying and addressing the OSS CPU and latency increase, but the issue provides no target threshold.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100