feat(storage): write pipeline robustness and edge-case handling for object_store backend
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Problem / Context
In #3165 (reviving #2257 for #2259), the foundational `ObjectStoreStorage::S3` backend is established. To ensure production robustness and performance in the write pipeline, several edge cases and optimizations should be addressed as follow-up work:
### Proposed Improvements
1. **Zero-Copy Writes in `ObjectStoreWriter`**:
- `FileWrite::write` receives owned `bytes::Bytes`.
- `object_store::WriteMultipart` provides a `put(Bytes)` method that buffers without slice copying, avoiding redundant `extend_from_slice` allocations.
2. **S3 URL Boundary Validation**:
- Empty bucket strings (e.g. `s3:///path/to/file` or `s3://`) currently pass initial URL host parsing as `""`. These should explicitly return `ErrorKind::DataInvalid`.
- Ensure paths without trailing slashes (`s3://bucket`) extract `bucket` and empty relative path cleanly.
3. **Stream Handling & State Machine Guards**:
- Validate `delete_stream` on empty streams (`futures::stream::empty()`) and ensure immediate error propagation on invalid URLs.
- Add unit test coverage for `ObjectStoreWriter` state guards (e.g., verifying `close()` and `write()` error handling after writer has been closed).
4. **Backpressure & Memory Throttling**:
- For high-throughput streaming, evaluate `WriteMultipart::wait_for_capacity` to throttle producers and prevent unbounded 5MB chunk allocations in memory ahead of S3 network uploads.
### Related
- #2258 (Umbrella: Implement Storage using object_store)
- #2259 (Implement ObjectStoreStorage::S3)
- #3165 (feat(storage): Implement ObjectStoreStorage::S3)
## AI Disclosure
Drafted with AI assistance; reviewed and verified by contributor.
Contributor guide
Research direction
Start with ObjectStoreStorage::S3, ObjectStoreWriter, FileWrite::write, and the WriteMultipart calls described in the issue. Check URL parsing for empty buckets and bucket-only paths, then inspect delete_stream and writer state handling before evaluating wait_for_capacity. Done means the listed edge cases return the expected errors, writes avoid redundant buffering, state-guard tests cover close/write failures, and streaming memory is appropriately bounded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100