apache / apache/opendal

new feature: in-memory distributed buffer as OpenDAL layer

Open
#7,236 1 comment 1 reaction 0 assignees View on GitHub
enhancement releases-note/feat
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

A layer that allows for multiple processes connected to the same underlying OpenDAL store to pick a single writer and route all writes to that process, transparently.

### Problem and Solution

I see it as a common issue that writes to object storage have to be coordinated.

Another common issue is that writes to object storage take significant time. A common solution to that is buffering in memory and replicating to multiple availability zones.

It feels (naively so perhaps) possible for an OpenDAL layer to transparently connect to other nodes in a cluster, relay writes to the leader, wait for the cluster to confirm.

### Additional Context

I believe Cloudflare Durable Objects use a similar design.

Originally I started thinking about this in the context of SlateDB, which I believe is considering using OpenDAL internally.

### Are you willing to contribute to the development of this feature?

- [x] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing OpenDAL's existing layer model and defining the cluster, leader-election, write-routing, replication, and failure semantics. Done should include an agreed scope and validation plan for a transparent distributed in-memory buffer.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.