quickwit-oss / quickwit-oss/quickwit

exit_status=downstream actor exited

Open
#5,867 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug
A clear and concise description of what the bug is.

I'm not very familiar with quickwit. Initially, the log ingestion worked fine, but after a while, I noticed a large number of errors. The data storage is on MinIO, and the indexing is on Postgres. The deployment was done using Helm and Terraform. The error messages are as follows:

2025-08-22T16:42:23.013Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=quickwit_indexing::actors::index_serializer::IndexSerializer-twilight-tj9q exit_status=success
2025-08-22T16:42:23.013Z  INFO quickwit_actors::spawn_builder: no more messages actor="Packager-withered-4LMj"
2025-08-22T16:42:23.013Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=Packager-withered-4LMj exit_status=success
2025-08-22T16:42:23.013Z  INFO quickwit_actors::spawn_builder: no more messages actor="IndexUploader-hidden-7bjg"
2025-08-22T16:42:23.013Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=IndexUploader-hidden-7bjg exit_status=success
2025-08-22T16:42:23.013Z  INFO quickwit_actors::spawn_builder: no more messages actor="quickwit_indexing::actors::sequencer::Sequencer<quickwit_indexing::actors::publisher::Publisher>-falling-n9hF"
2025-08-22T16:42:23.013Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=quickwit_indexing::actors::sequencer::Sequencer<quickwit_indexing::actors::publisher::Publisher>-falling-n9hF exit_status=success
2025-08-22T16:42:23.013Z  INFO quickwit_actors::spawn_builder: no more messages actor="Publisher-spring-zHGO"
2025-08-22T16:42:23.013Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=Publisher-spring-zHGO exit_status=success
2025-08-22T16:42:23.022Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=quickwit_indexing::actors::doc_processor::DocProcessor-shy-AgX2 exit_status=downstream actor exited
2025-08-22T16:42:23.022Z ERROR quickwit_actors::actor_context: exit activating-kill-switch actor=quickwit_indexing::actors::doc_processor::DocProcessor-shy-AgX2 exit_status=DownstreamClosed
2025-08-22T16:42:23.023Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=SourceActor-withered-UTbm exit_status=killed
2025-08-22T16:42:23.437Z ERROR quickwit_actors::actor_handle: actor-exit-without-success actor="SourceActor-withered-UTbm"
2025-08-22T16:42:23.437Z ERROR quickwit_actors::actor_handle: actor-exit-without-success actor="quickwit_indexing::actors::doc_processor::DocProcessor-shy-AgX2"
2025-08-22T16:42:23.437Z ERROR quickwit_indexing::actors::indexing_pipeline: Indexing pipeline failure. pipeline_id=IndexingPipelineId { node_id: "vt-quickwit-indexer-0", index_uid: IndexUid { index_id: "files", incarnation_id: Ulid(2121395130011908575915230040062802881) }, source_id: "_ingest-api-source", pipeline_uid: Pipeline(01K2973MTRGM5EXAFVFVA4KD1X) } generation=522937 healthy_actors=["Indexer-crimson-j5QY"] failed_or_unhealthy_actors=["SourceActor-withered-UTbm", "quickwit_indexing::actors::doc_processor::DocProcessor-shy-AgX2"] success_actors=["quickwit_indexing::actors::index_serializer::IndexSerializer-twilight-tj9q", "Packager-withered-4LMj", "IndexUploader-hidden-7bjg", "quickwit_indexing::actors::sequencer::Sequencer<quickwit_indexing::actors::publisher::Publisher>-falling-n9hF", "Publisher-spring-zHGO"]

here is the config:

{
            name = "config.default_index_root_uri"
            value = "s3://quickwit/indexes"
        },
        {
            name = "config.metastore_uri"
            value = "postgres://${local.postgres_user}:quickwitStrong12%23@vt-quickwit-minio-postgresql/${local.postgres_db}"
        },
        {
            # 
            name = "config.storage.s3.endpoint"
            value = "http://4.5.0.3:9000"
        },
        {
            name = "config.storage.s3.flavor"
            value = "minio"
        },
        {
            name = "config.storage.s3.access_key_id"
            value = local.access_key_id
        },
        {
            name = "config.storage.s3.force_path_style_access"
            value = true
        },
        {
            name = "config.storage.s3.secret_access_key"
            value = local.secret_access_key
        },
        {
            name = "config.storage.s3.region"
            value = "us-east-1"
        },
        {
            name = "ingress.enabled"
            value = true
        },
        {
            name = "ingress.ingressClassName"
            value = "nginx"
        },
        {
            name = "ingress.hostname"
            value = "quickwit.local"
        },
        {
            name = "config.indexer.split_store_max_num_bytes"
            value = "200G"
        },
        {
            name = "indexer.replicaCount"
            value = 3
        },
        {
            name = "config.indexer.split_store_max_num_splits"
            value = "10000"
        },
        {
            name = "config.ingest_api.max_queue_memory_usage"
            value = "20GiB"
        },
        {
            name = "config.ingest_api.max_queue_disk_usage"
            value = "30GiB"
        },
        {
            name = "config.searcher.fast_field_cache_capacity"
            value = "10G"
        },
        {
            name = "config.searcher.split_footer_cache_capacity"
            value = "10G"
        },
        {
            name = "config.searcher.max_num_concurrent_split_streams"
            value = "100"
        }

       

  1. Output of quickwit --version
    0.8.1
  2. The index_config.yaml

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied indexing-pipeline and actor-exit logs, focusing on SourceActor, DocProcessor, and the downstream actor chain. Then verify the Quickwit 0.8.1 configuration for the Postgres metastore and S3 storage in the Helm/Terraform deployment. Done means identifying the cause of the shutdown and documenting a reproducible fix or the missing diagnostic data.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, postgres, rust, terraform
Domain
backend, cloud, databases, devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.