influxdata / influxdata/influxdb

Last value Cache - Increase in outliers beyond 16 threads for 100 series

Open
#25,562 29 comments 0 reactions 0 assignees View on GitHub
kind/perf v3
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

### Increase in Outliers beyond 16 thread concurrency (Last value cache)

- For concurrency threads greater than 16, we are observing more outliers nearly 5x-10x of the typical latency thus impacting the P95 numbers
- CPU usage was less than 20% and Memory consumption was less than 25%
- This observation and pattern looks like there are some restrictions/limitations that are paving way for outliers of latencies.

Could there be a wait happening on some internal resources?

#### Evidence

image
image
image

Note: How we capture latency (P95 reported) is by having backgrounded threads which are 12, 14, 16 etc. and collect the metrics from just one. This shows on concurrent load, how a particular user observes performance. Stating that, QPS could have been impacted by the outliers observed.

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Reproduce the Last value Cache workload with 12, 14, and 16+ backgrounded threads, measuring P95, QPS, CPU, and memory, then investigate possible waits or resource limits. Done means identifying the cause of the latency outliers and showing their impact under the reported concurrency.

Written by the indexing model from the issue text.

Assessment

Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.