influxdata / influxdata/influxdb
[v3] Lift concurrent query semaphore limit by default
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
### Problem statement
The query executor has a hard-coded limit on the concurrent query semaphore of `10`: https://github.com/influxdata/influxdb/blob/ef3599d7ce0139624f65cedb9e9b84b0d275f3a5/influxdb3/src/commands/serve.rs#L520
This should be higher by default, and also be configurable.
### Proposed solution
Completely remove the limit by default by setting the default to the max for `tokio`'s semaphore (see [`Semaphore::MAX_PERMITS`](https://docs.rs/tokio/latest/tokio/sync/struct.Semaphore.html#associatedconstant.MAX_PERMITS)).
Make the value configurable in the CLI arguments with a `--max-concurrent-queries` argument.
### Alternatives
Don't allow a configurable limit, and just set it internally to `Semaphore::MAX_PERMITS`.
### Additional context
The value of `10` likely came from IOx, where the default value for that semaphore in the querier is chosen as `10`. There, however, that is being set on a _per-querier_ basis, so they can support more concurrent queries in practice via multiple nodes.
See also: https://github.com/influxdata/influxdb/issues/25615
Contributor guide
Research direction
Read influxdb3/src/commands/serve.rs at the referenced semaphore setup, then trace the CLI argument definitions. Check issue #25615 for related context. Done means the default is Tokio’s maximum and --max-concurrent-queries configures the limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, database
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 67/100