quickwit-oss / quickwit-oss/quickwit
Add a way to have concurrent actors
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Right now we do not have the possibility to have concurrent actors.
It is lame especially in the case of Uploader (an async actors), because it makes it difficult for us to use our bandwidth efficiently by concurrently uploading splits. The current workaround is relying on a semaphore, which is too convoluted.
For the merge executor too, (a sync actor) we may need for more settings to have more than one thread working on merging splits.
Solution
Workers have a mailbox composed of :
- a regular high priority channel (for command and scheduled message)
- a shared common low priority channel, and consumes it concurrently.
A parallel actor is in charge of the supervision of these actors.
Commands sent to the parallel actor are dispatched to all of the worker actors, while message are
send to the shared low priority channel, and consumed concurrently.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing actor, mailbox, Uploader, and merge executor implementations. Review how the current semaphore workaround handles concurrent uploads, then assess the proposed worker and parallel-actor supervision model. Done means actors can process work concurrently while commands remain dispatched to all workers and both uploader and merge workloads support multiple workers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100