quickwit-oss / quickwit-oss/quickwit
Support for "remote" (EBS, NFS) file systems as data dir
Open
Nobody has claimed this yet.
bug
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Typically, users use /qwdata as a mount point for those file systems. Then, a Quickwit node believes the files stored in the data dir are local, and access is not shared. This creates some issues:
- multiple nodes are going to write at the same location, for instance in
/qwdata/cache/splits,/qwdata/queues, step on each other's toes, and eventually lose data and fail. - caching files in the data dir (
SplitStore,MergeSplitDownloader) no longer works - actors time out because some operations take seconds instead of milliseconds.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the data directory is used for split caches, queues, SplitStore, and MergeSplitDownloader. Confirm which operations assume local storage and which actor timeouts are affected. Done means remote EBS/NFS data directories are supported without nodes sharing unsafe state, broken caching, data loss, or timing failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100