influxdata / influxdata/docs-v2
Clustered: modifying querier RAM allocation
- Dominant language
- JavaScript
- Stars
- 82
- Forks
- 326
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 82
Description
INFLUXDB_IOX_EXEC_MEM_POOL_BYTES is [mentioned, but only as an example](https://docs.influxdata.com/influxdb3/clustered/admin/env-vars/#view-example-of-environment-variables-in-all-components). It really shouldn't be edited in isolation. We probably should have some guidance around the topic of adjusting querier RAM allocation:
-----
Why adjust the RAM allocation? The only common use case: Your queriers have underutilized memory and your workload could benefit from caching more parquet (high query latency as a result of continuous or frequent object store accesses). Only adjust in response to empirical evidence. In this case you may want to decrease INFLUXDB_IOX_EXEC_MEM_POOL_BYTES, and increase INFLUXDB_IOX_RAM_POOL_DATA_BYTES by the same amount.
Current allocation of the queriers' memory allocation (requests):
- 50% to INFLUXDB_IOX_EXEC_MEM_POOL_BYTES (query execution)
- 35% to INFLUXDB_IOX_RAM_POOL_DATA_BYTES (in-memory parquet cache)
- 5% to INFLUXDB_IOX_RAM_POOL_METADATA_BYTES (metadata)
- (10% remaining for everything else)
Why not adjust the allocation: the defaults work well for many workloads and ensure stability. To avoid instability, avoid:
- picking numbers that exceed the amount of memory the querier has
- cutting in to the 10% remainder
- reducing the metadata allocation
- eliminating too much headroom in the execution pool; the defaults are intentionally generous here to avoid query failures or OOMs (especially during exceptional circumstances such as increased load from the clients or decreased capacity during rollouts or node replacement)
- speculative changes based on a short period of observation (make you have a good sense of what worst case load looks like)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked environment-variable documentation page at docs.influxdata.com/influxdb3/clustered/admin/env-vars/#view-example-of-environment-variables-in-all-components. Add the querier RAM allocation guidance from the issue, including the allocation breakdown, adjustment criteria, and stability warnings. Done means readers can understand when and how to adjust the two pools without editing the execution pool in isolation.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation, infrastructure
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100