quickwit-oss / quickwit-oss/quickwit

Prometheus metrics for ingest wal usage are not working

Open
#5,547 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug

Both quickwit_ingest_wal_disk_used_bytes and quickwit_ingest_wal_memory_used_bytes are not working as expected. quickwit_ingest_wal_disk_used_bytes displays a constant value of 134217728 (max queue disk usage is set to 32GB and total size of disk is 250G). quickwit_ingest_wal_memory_used_bytes always report 0.

Steps to reproduce (if applicable)
I'm using default Prometheus scraping configuration provided by the helm chart. These are my ingest_api values:

  ingest_api:
    max_queue_memory_usage: 4GiB
    max_queue_disk_usage: 32GiB

Expected behavior
I expect both metrics to report WAL usage for both disk and memory.

Would also be great to have metrics to show max_queue_disk_usage and max_queue_mem_usage config setting.

Configuration:
Version: v0.8.1

node.yaml:

data_dir: /quickwit/qwdata
default_index_root_uri: s3://prod-<redacted>-quickwit/indexes
gossip_listen_port: 7282
grpc:
  max_message_size: 80 MiB
indexer:
  enable_otlp_endpoint: true
ingest_api:
  max_queue_disk_usage: 32GiB
  max_queue_memory_usage: 4GiB
listen_address: 0.0.0.0
metastore:
  postgres:
    acquire_connection_timeout: 30s
    idle_connection_timeout: 1h
    max_connection_lifetime: 1d
    max_connections: 50
    min_connections: 10
storage:
  s3:
    region: us-east-1
version: 0.8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the quickwit_ingest_wal_disk_used_bytes and quickwit_ingest_wal_memory_used_bytes metrics through the ingest API and the default Helm Prometheus scraping configuration. Reproduce the reported values with max_queue_memory_usage set to 4GiB and max_queue_disk_usage set to 32GiB; done means both metrics reflect WAL disk and memory usage, with any added limit metrics matching the configured values.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, prometheus, rust
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.