quickwit-oss / quickwit-oss/quickwit

Does not consume queue if indexer > 1

Open
#5,868 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug
When we put 1 more indexer in production, current source stop to consume

Steps to reproduce (if applicable)
Steps to reproduce the behavior:

  1. deploy 1 indexer
  2. create 1 index with 1 Pulsar topic
  3. configure a new index source to consume the topic
  4. Add a second indexer in production

Expected behavior
all is working as previously but with HA

Current behaviour

2025-08-26T15:03:24.497Z ERROR pulsar::consumer::engine: Broker notification of closed consumer 1: [1 - quickwit-orga_85.traces:01JZ36YXRFBSGD1QEX0E2R2XMZ-pulsar.traces(quickwit): persistent://user_32/pulsar_82/orga_85.traces-partition-0]
2025-08-26T15:03:24.497Z  WARN pulsar::consumer::engine: rx terminated
2025-08-26T15:03:24.585Z  WARN pulsar::retry_op: Retry #0 -> connecting consumer 1 using connection ff080b03-686a-4f1a-bd13-3fc2db6fbe29 to broker pulsar+ssl://materiamq.eu-fr-1.services.clever-cloud.com:6651 to topic persistent://user_32/pulsar_82/orga_85.traces-partition-0
2025-08-26T15:03:25.223Z ERROR quickwit_indexing::actors::indexing_pipeline: error while spawning indexing pipeline, retrying after some time error=failed to create source `_ingest-api-source` of type `ingest API v1`. Cause: internal error: could not find checkpoint for index `orga_85.traces:01JZ36YXRFBSGD1QEX0E2R2XMZ` and source `_ingest-api-source`; cause: ``
Caused by:
    internal error: could not find checkpoint for index `orga_85.traces:01JZ36YXRFBSGD1QEX0E2R2XMZ` and source `_ingest-api-source`; cause: `` retry_count=0 retry_delay=2s
2025-08-26T15:03:25.229Z  WARN pulsar::consumer::engine: rx terminated
2025-08-26T15:03:25.230Z ERROR quickwit_actors::actor_context: exit activating-kill-switch actor=SourceActor-rough-R5s6 exit_status=DownstreamClosed
2025-08-26T15:03:25.230Z  WARN pulsar::consumer::engine: rx terminated

Configuration:
Please provide:

  1. Output of quickwit --version => Quickwit 0.8.0 (x86_64-unknown-linux-gnu 2025-02-06T15:26:42Z 7bc495a)
  2. The index_config.yaml
# ============================ Node Configuration ==============================
#
# Website: https://quickwit.io
# Docs: https://quickwit.io/docs/configuration/node-config
#
# Configure AWS credentials: https://quickwit.io/docs/guides/aws-setup#aws-credentials
#
# -------------------------------- General settings --------------------------------
#
# Config file format version.
#
version: 0.7

# Unique identifier of the cluster the node will be joining.
# Clusters sharing the same network should use distinct cluster IDs.
#
cluster_id: qw-c1

# Node ID. Must be unique within a cluster. If not set, a random node ID is generated on each startup.
#
node_id: qw-c1-n1


# https://github.com/quickwit-oss/quickwit/blob/main/quickwit/quickwit-config/src/service.rs#L49
#
enabled_services:
  - control_plane
  - indexer
  - janitor
  - metastore
  - searcher

# Quickwit opens three sockets.
# - for its HTTP server, hosting the UI and the REST API (TCP)
# - for its gRPC service (TCP)
# - for its Gossip cluster membership service (UDP)
#
# All three services are bound to the same host and a different port. The host can be an IP address or a hostname.
#
# Default HTTP server host is `127.0.0.1` and default HTTP port is 7280.
# The default host value was chosen to avoid exposing the node to the open-world without users' explicit consent.
# This allows for testing Quickwit in single-node mode or with multiple nodes running on the same host and listening
# on different ports. However, in cluster mode, using this value is never appropriate because it causes the node to
# ignore incoming traffic.
# There are two options to set up a node in cluster mode:
#   1. specify the node's hostname or IP
#   2. pass `0.0.0.0` and let Quickwit do its best to discover the node's IP (see `advertise_address`)
#
listen_address: 192.168.0.1

rest:
  listen_port: 7280
  cors_allow_origins:
    - "http://localhost:3000"
#   extra_headers:
#     x-header-1: header-value-1
#     x-header-2: header-value-2

grpc:
  max_message_size: 10 MiB
grpc_listen_port: 7281

gossip_listen_port: 7280

# IP address advertised by the node, i.e. the IP address that peer nodes should use to connect to the node for RPCs.
# The environment variable `QW_ADVERTISE_ADDRESS` can also be used to override this value.
# The default advertise address is `listen_address`. If `listen_address` is unspecified (`0.0.0.0`),
# Quickwit attempts to sniff the node's IP by scanning the available network interfaces.
advertise_address: 192.168.0.1

# In order to join a cluster, one needs to specify a list of
# seeds to connect to. If no port is specified, Quickwit will assume
# the seeds are using the same port as the current node gossip port.
# By default, the peer seed list is empty.
#
peer_seeds: 
  - qw-c1-n2:7280

#   - quickwit-searcher-0.local
#   - quickwit-searcher-1.local:10000
#
# Path to directory where temporary data (caches, intermediate indexing data structures)
# is stored. Defaults to `./qwdata`.
#
data_dir: /data/qwdata


# Metastore URI. Defaults to `data_dir/indexes#polling_interval=30s`,
# which is a file-backed metastore and mostly convenient for testing. A cluster would
# require a metastore backed by Amzon S3 or PostgreSQL.
#
# metastore_uri: s3://your-bucket/indexes
metastore_uri: postgresql://REDACTED

# When using a file-backed metastore, the state of the metastore will be cached forever.
# If you are indexing and searching from different processes, it is possible to periodically
# refresh the state of the metastore on the searcher using the `polling_interval` hashtag.
#
# metastore_uri: s3://your-bucket/indexes#polling_interval=30s

# Default index root URI, which defines where index data (splits) is stored,
# following the scheme `{default_index_root_uri}/{index-id}`. Defaults to `{data_dir}/indexes`.
#
# default_index_root_uri: s3://your-bucket/indexes

# -------------------------------- Storage settings --------------------------------

# Hardcoding credentials into configuration files is not secure and strongly
# discouraged. Prefer the alternative authentication methods that your storage
# backend may provide.
#
storage:
#   azure:
#     account: ${QW_AZURE_STORAGE_ACCOUNT}
#     access_key: ${QW_AZURE_STORAGE_ACCESS_KEY}
#
  s3:
    access_key_id: "REDACTED"
    secret_access_key: "REDACTED"
    region: default
    endpoint: https://cellar-c2.services.clever-cloud.com
    force_path_style_access: ${QW_S3_FORCE_PATH_STYLE_ACCESS:-false}
    disable_multi_object_delete: false
    disable_multipart_upload: false
#
# -------------------------------- Metastore settings --------------------------------

metastore:
  postgres:
    max_num_connections: 10

# -------------------------------- Indexer settings --------------------------------

indexer:
  enable_otlp_endpoint: True
  split_store_max_num_bytes: 100G # TODO match data disk size
  split_store_max_num_splits: 1000
  max_concurrent_split_uploads: 12


# -------------------------------- Ingest API settings ------------------------------

ingest_api:
  max_queue_memory_usage: 2GiB
  max_queue_disk_usage: 4GiB

# -------------------------------- Searcher settings --------------------------------

searcher:
  fast_field_cache_capacity: 1G
  split_footer_cache_capacity: 500M
  max_num_concurrent_split_streams: 100
  partial_request_cache_capacity: 64M
  max_num_concurrent_split_searches: 100

# -------------------------------- Jaeger settings --------------------------------

jaeger:
  enable_endpoint: True

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the indexer and Pulsar consumer errors in the report, then reproduce the four deployment steps using Quickwit 0.8.0 and the supplied configuration. Compare behavior before and after adding the second indexer, including the checkpoint error for _ingest-api-source. Done means the source continues consuming in HA without the reported consumer closure or missing-checkpoint failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.