new feature: provide per-operation concurrent limit
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
Similar to https://github.com/apache/opendal/issues/7168, I'd like to have per-operation concurrent limit.
### Problem and Solution
opendal natively supports `ConcurrentLimitLayer`, if I don't read it wrongly, all operations (whether it's IO like read/write, or metadata only like stats/list), they all acquire permit on the same semaphore. Code reference below:
- https://github.com/apache/opendal/blob/6100a695de1ece1e2f986bbc5e51b9b056b3b97c/core/layers/concurrent-limit/src/lib.rs#L258
- https://github.com/apache/opendal/blob/6100a695de1ece1e2f986bbc5e51b9b056b3b97c/core/layers/concurrent-limit/src/lib.rs#L264
- and many others
Different operations usually hit different storage backends (i.e., filesystem vs TP database), and showing different workload characteristics (i.e., IOPS-oriented vs throughput-oriented), I think it's better to provide a per-operation concurrent limit.
For me specifically, I met severe read and write server overload issue, so client-side need to have our own gating for in-flight requests for RW only, but not for metadata operations.
### Additional Context
I could implement my own layer to achieve per-operation config, but I think it should be a useful feature to support in opendal core.
### Are you willing to contribute to the development of this feature?
- [ ] Yes, I am willing to contribute to the development of this feature.
Contributor guide
Research direction
Start by reading core/layers/concurrent-limit/src/lib.rs at the referenced lines and compare the proposed behavior with apache/opendal#7168. Determine how per-operation limits should distinguish read/write requests from metadata operations. Done means OpenDAL core has a defined, usable per-operation concurrency-limit feature that supports the stated read/write gating without applying the same limit to metadata operations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100