influxdata / influxdata/influxdb
Last value Cache - Increase in outliers beyond 16 threads for 100 series
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
### Increase in Outliers beyond 16 thread concurrency (Last value cache)
- For concurrency threads greater than 16, we are observing more outliers nearly 5x-10x of the typical latency thus impacting the P95 numbers
- CPU usage was less than 20% and Memory consumption was less than 25%
- This observation and pattern looks like there are some restrictions/limitations that are paving way for outliers of latencies.
Could there be a wait happening on some internal resources?
#### Evidence
Note: How we capture latency (P95 reported) is by having backgrounded threads which are 12, 14, 16 etc. and collect the metrics from just one. This shows on concurrent load, how a particular user observes performance. Stating that, QPS could have been impacted by the outliers observed.
Contributor guide
Research direction
No source file or test is named. Reproduce the Last value Cache workload with 12, 14, and 16+ backgrounded threads, measuring P95, QPS, CPU, and memory, then investigate possible waits or resource limits. Done means identifying the cause of the latency outliers and showing their impact under the reported concurrency.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100