quickwit-oss / quickwit-oss/quickwit

searcher/indexer join/leave the cluster frequently

Open
#4,313 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug
I deploy quickwit on EKS. HTTP probes of indexers and searchers are failed with statuscode 503. Both indexers and searches output lots of the following logs repeatly:

WARN quickwit_serve: Metastore service is unavailable. metastore_uri=grpc://metastore.service.cluster error=The metastore service is unavailable.

metastore output lots of the following logs:

2023-12-21T11:59:06.635Z  INFO quickwit_cluster::change: Node `quickwit-indexer-1` has left the cluster. cluster_id=quickwit node_id=quickwit-indexer-1
2023-12-21T11:59:07.636Z  INFO quickwit_cluster::change: Node `quickwit-indexer-1` has joined the cluster. cluster_id=quickwit node_id=quickwit-indexer-1
2023-12-21T11:59:13.635Z  INFO quickwit_cluster::change: Node `quickwit-indexer-2` has left the cluster. cluster_id=quickwit node_id=quickwit-indexer-2
2023-12-21T11:59:14.634Z  INFO quickwit_cluster::change: Node `quickwit-indexer-2` has joined the cluster. cluster_id=quickwit node_id=quickwit-indexer-2
2023-12-21T11:59:15.636Z  INFO quickwit_cluster::change: Node `quickwit-searcher-2` has left the cluster. cluster_id=quickwit node_id=quickwit-searcher-2
2023-12-21T11:59:26.649Z  INFO quickwit_cluster::change: Node `quickwit-searcher-2` has joined the cluster. cluster_id=quickwit node_id=quickwit-searcher-2
2023-12-21T11:59:55.637Z  INFO quickwit_cluster::change: Node `quickwit-indexer-1` has left the cluster. cluster_id=quickwit node_id=quickwit-indexer-1
2023-12-21T11:59:57.637Z  INFO quickwit_cluster::change: Node `quickwit-indexer-1` has joined the cluster. cluster_id=quickwit node_id=quickwit-indexer-1
2023-12-21T12:00:46.634Z  INFO quickwit_cluster::change: Node `quickwit-searcher-2` has left the cluster. cluster_id=quickwit node_id=quickwit-searcher-2
2023-12-21T12:00:49.635Z  INFO quickwit_cluster::change: Node `quickwit-indexer-2` has left the cluster. cluster_id=quickwit node_id=quickwit-indexer-2
2023-12-21T12:00:50.638Z  INFO quickwit_cluster::change: Node `quickwit-indexer-2` has joined the cluster. cluster_id=quickwit node_id=quickwit-indexer-2
2023-12-21T12:01:06.636Z  INFO quickwit_cluster::change: Node `quickwit-searcher-0` has joined the cluster. cluster_id=quickwit node_id=quickwit-searcher-0
2023-12-21T12:01:06.636Z  INFO quickwit_cluster::change: Node `quickwit-searcher-2` has joined the cluster. cluster_id=quickwit node_id=quickwit-searcher-2
2023-12-21T12:01:26.637Z  INFO quickwit_cluster::change: Node `quickwit-searcher-0` has left the cluster. cluster_id=quickwit node_id=quickwit-searcher-0
2023-12-21T12:01:27.634Z  INFO quickwit_cluster::change: Node `quickwit-indexer-1` has left the cluster. cluster_id=quickwit node_id=quickwit-indexer-1
2023-12-21T12:01:28.634Z  INFO quickwit_cluster::change: Node `quickwit-indexer-1` has joined the cluster. cluster_id=quickwit node_id=quickwit-indexer-1
2023-12-21T12:01:31.636Z  INFO quickwit_cluster::change: Node `quickwit-searcher-2` has left the cluster. cluster_id=quickwit node_id=quickwit-searcher-2
2023-12-21T12:01:34.635Z  INFO quickwit_cluster::change: Node `quickwit-searcher-2` has joined the cluster. cluster_id=quickwit node_id=quickwit-searcher-2
2023-12-21T12:02:33.633Z  INFO quickwit_cluster::change: Node `quickwit-indexer-1` has left the cluster. cluster_id=quickwit node_id=quickwit-indexer-1
2023-12-21T12:02:34.633Z  INFO quickwit_cluster::change: Node `quickwit-indexer-1` has joined the cluster. cluster_id=quickwit node_id=quickwit-indexer-1
2023-12-21T12:03:15.636Z  INFO quickwit_cluster::change: Node `quickwit-searcher-2` has left the cluster. cluster_id=quickwit node_id=quickwit-searcher-2
2023-12-21T12:03:18.634Z  INFO quickwit_cluster::change: Node `quickwit-searcher-2` has joined the cluster. cluster_id=quickwit node_id=quickwit-searcher-2
2023-12-21T12:03:26.634Z  INFO quickwit_cluster::change: Node `quickwit-indexer-1` has left the cluster. cluster_id=quickwit node_id=quickwit-indexer-1
2023-12-21T12:03:27.634Z  INFO quickwit_cluster::change: Node `quickwit-indexer-1` has joined the cluster. cluster_id=quickwit node_id=quickwit-indexer-1

It seems searches and indexers are joining/leaving the cluster frequently. Quickwit search rest API is not working. I don't if
ingest API is working because I can not get data from search API.

Steps to reproduce (if applicable)
I have no idea about how to reproduce.

Expected behavior
How to fix it?

Configuration:
Please provide:

  1. Output of quickwit --version
Quickwit v0.6.5 (5cf786d 2023-12-11T13:37:05Z)
  1. The index_config.yaml
version: 0.6

index_id: my-index

doc_mapping:
  mode: strict
  field_mappings:
    - name: trace_id
      type: text
      fast: true
    - name: trace_state
      type: text
      indexed: false
    - name: service_name
      type: text
      tokenizer: raw
    - name: resource_attributes
      type: json
      tokenizer: raw
    - name: resource_dropped_attributes_count
      type: u64
      indexed: false
    - name: scope_name
      type: text
      indexed: true
    - name: scope_version
      type: text
      indexed: false
    - name: scope_attributes
      type: json
      indexed: false
    - name: scope_dropped_attributes_count
      type: u64
      indexed: false
    - name: span_id
      type: text
      tokenizer: raw
    - name: parent_span_id
      type: text
      fast: true
      tokenizer: raw
    - name: span_kind
      type: u64
    - name: span_name
      type: text
      tokenizer: raw
    - name: span_start_timestamp_nanos
      type: datetime
      input_formats: [unix_timestamp]
      output_format: unix_timestamp_nanos
      indexed: false
      fast: true
      precision: milliseconds
    - name: span_end_timestamp_nanos
      type: datetime
      input_formats: [unix_timestamp]
      output_format: unix_timestamp_nanos
      indexed: false
      fast: false
    - name: span_duration_millis
      type: u64
      indexed: false
      fast: true
      stored: false
    - name: span_attributes
      type: json
      tokenizer: en_stem
      record: position
    - name: span_dropped_attributes_count
      type: u64
      indexed: false
    - name: span_dropped_events_count
      type: u64
      indexed: false
    - name: span_dropped_links_count
      type: u64
      indexed: false
    - name: span_status
      type: json
      indexed: true
    - name: events
      type: array<json>
      tokenizer: raw
    - name: event_names
      type: array<text>
      tokenizer: default
      record: position
      stored: false
    - name: links
      type: array<json>
      tokenizer: raw

  timestamp_field: span_start_timestamp_nanos

  # partition_key: hash_mod(service_name, 100)
  # tag_fields: [service_name]

indexing_settings:
  commit_timeout_secs: 5

search_settings:
  default_search_fields: []

retention:
  period: 5 days
  schedule: daily

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file, test, or entry point is named. Begin with the EKS deployment, HTTP 503 symptoms, and metastore join/leave logs; done means identifying why nodes repeatedly leave and rejoin and restoring successful health probes and search API operation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, rust
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.