new feature: in-memory distributed buffer as OpenDAL layer
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
A layer that allows for multiple processes connected to the same underlying OpenDAL store to pick a single writer and route all writes to that process, transparently.
### Problem and Solution
I see it as a common issue that writes to object storage have to be coordinated.
Another common issue is that writes to object storage take significant time. A common solution to that is buffering in memory and replicating to multiple availability zones.
It feels (naively so perhaps) possible for an OpenDAL layer to transparently connect to other nodes in a cluster, relay writes to the leader, wait for the cluster to confirm.
### Additional Context
I believe Cloudflare Durable Objects use a similar design.
Originally I started thinking about this in the context of SlateDB, which I believe is considering using OpenDAL internally.
### Are you willing to contribute to the development of this feature?
- [x] Yes, I am willing to contribute to the development of this feature.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reviewing OpenDAL's existing layer model and defining the cluster, leader-election, write-routing, replication, and failure semantics. Done should include an agreed scope and validation plan for a transparent distributed in-memory buffer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100