influxdata / influxdata/influxdb

[v3] Lift concurrent query semaphore limit by default

Open
#25,627 4 comments 0 reactions 0 assignees View on GitHub
v3
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

### Problem statement

The query executor has a hard-coded limit on the concurrent query semaphore of `10`: https://github.com/influxdata/influxdb/blob/ef3599d7ce0139624f65cedb9e9b84b0d275f3a5/influxdb3/src/commands/serve.rs#L520

This should be higher by default, and also be configurable.

### Proposed solution

Completely remove the limit by default by setting the default to the max for `tokio`'s semaphore (see [`Semaphore::MAX_PERMITS`](https://docs.rs/tokio/latest/tokio/sync/struct.Semaphore.html#associatedconstant.MAX_PERMITS)).

Make the value configurable in the CLI arguments with a `--max-concurrent-queries` argument.

### Alternatives

Don't allow a configurable limit, and just set it internally to `Semaphore::MAX_PERMITS`.

### Additional context

The value of `10` likely came from IOx, where the default value for that semaphore in the querier is chosen as `10`. There, however, that is being set on a _per-querier_ basis, so they can support more concurrent queries in practice via multiple nodes.

See also: https://github.com/influxdata/influxdb/issues/25615

Contributor guide

Open the contributing guide

Research direction

Read influxdb3/src/commands/serve.rs at the referenced semaphore setup, then trace the CLI argument definitions. Check issue #25615 for related context. Done means the default is Tokio’s maximum and --max-concurrent-queries configures the limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, database
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
67/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.