influxdata / influxdata/influxdb
Slowness when doing basic queries
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
### Discussed in https://github.com/influxdata/influxdb/discussions/23690
Originally posted by **jeremyaouad** August 31, 2022
I was testing with Influx 2.3 container.
I have a set of 80 metrics, emitted every 10 seconds, for about 500 processes => cardinality of 500*80 = 40000.
I didn't put any limits on the influxdb container cpu and memory.
The machine I'm testing on has 28 cores and 500GB ram. I'm using the ram as storage for influxdb (bbolt mounted on a tmpfs volume), with a read/write speed of 3 GB/sec.
I wrote 12 very basic queries of the sort "select * from x where y = z" in a file and launched them periodically, every 10 sec, and timed the duration of the execution. In the query, I set the range for the past 10 seconds. The returned data set would be between 200 and 400 records.
I observed the following:
- For the write performance, the cpu went up to 5-10% max. Memory was also good.
- For the read performance, whenever the script was launched, the cpu usage went up to 12 cores, memory was ok. However, the execution time kept on increasing from a couple of seconds to 2 min as time passed. During the execution time, the cpu usage would be 12 cores.
- What puzzled me more is the following: I stopped emitting metrics in influxdb, I queried for the past 10 seconds, and I was still getting a 2 min query time.
I also launched the queries with the profiler output. The execution time was taking all of the time. Specifically this:
influxdb.readWindowAggregateSource,ReadWindowAggregateByTime11
Is there something I'm missing?
Thanks.
Contributor guide
Research direction
Reproduce the report using the InfluxDB 2.3 container, 40,000-series cardinality, and repeated 10-second queries, then inspect the profiler output around influxdb.readWindowAggregateSource and ReadWindowAggregateByTime11. Done should include an identified cause for the increasing query time and evidence that the basic queries no longer take minutes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100