quickwit-oss / quickwit-oss/quickwit
Prometheus metrics for ingest wal usage are not working
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Describe the bug
Both quickwit_ingest_wal_disk_used_bytes and quickwit_ingest_wal_memory_used_bytes are not working as expected. quickwit_ingest_wal_disk_used_bytes displays a constant value of 134217728 (max queue disk usage is set to 32GB and total size of disk is 250G). quickwit_ingest_wal_memory_used_bytes always report 0.
Steps to reproduce (if applicable)
I'm using default Prometheus scraping configuration provided by the helm chart. These are my ingest_api values:
ingest_api:
max_queue_memory_usage: 4GiB
max_queue_disk_usage: 32GiB
Expected behavior
I expect both metrics to report WAL usage for both disk and memory.
Would also be great to have metrics to show max_queue_disk_usage and max_queue_mem_usage config setting.
Configuration:
Version: v0.8.1
node.yaml:
data_dir: /quickwit/qwdata
default_index_root_uri: s3://prod-<redacted>-quickwit/indexes
gossip_listen_port: 7282
grpc:
max_message_size: 80 MiB
indexer:
enable_otlp_endpoint: true
ingest_api:
max_queue_disk_usage: 32GiB
max_queue_memory_usage: 4GiB
listen_address: 0.0.0.0
metastore:
postgres:
acquire_connection_timeout: 30s
idle_connection_timeout: 1h
max_connection_lifetime: 1d
max_connections: 50
min_connections: 10
storage:
s3:
region: us-east-1
version: 0.8
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the quickwit_ingest_wal_disk_used_bytes and quickwit_ingest_wal_memory_used_bytes metrics through the ingest API and the default Helm Prometheus scraping configuration. Reproduce the reported values with max_queue_memory_usage set to 4GiB and max_queue_disk_usage set to 32GiB; done means both metrics reflect WAL disk and memory usage, with any added limit metrics matching the configured values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, prometheus, rust
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100