quickwit-oss / quickwit-oss/quickwit

Add a way to have concurrent actors

Open
#517 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

backlog enhancement low-priority
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Right now we do not have the possibility to have concurrent actors.

It is lame especially in the case of Uploader (an async actors), because it makes it difficult for us to use our bandwidth efficiently by concurrently uploading splits. The current workaround is relying on a semaphore, which is too convoluted.

For the merge executor too, (a sync actor) we may need for more settings to have more than one thread working on merging splits.

Solution

Workers have a mailbox composed of :

  • a regular high priority channel (for command and scheduled message)
  • a shared common low priority channel, and consumes it concurrently.

A parallel actor is in charge of the supervision of these actors.

Commands sent to the parallel actor are dispatched to all of the worker actors, while message are
send to the shared low priority channel, and consumed concurrently.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing actor, mailbox, Uploader, and merge executor implementations. Review how the current semaphore workaround handles concurrent uploads, then assess the proposed worker and parallel-actor supervision model. Done means actors can process work concurrently while commands remain dispatched to all workers and both uploader and merge workloads support multiple workers.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.