apache / apache/iceberg-rust

feat(storage): write pipeline robustness and edge-case handling for object_store backend

Open
#3,166 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Problem / Context

In #3165 (reviving #2257 for #2259), the foundational `ObjectStoreStorage::S3` backend is established. To ensure production robustness and performance in the write pipeline, several edge cases and optimizations should be addressed as follow-up work:

### Proposed Improvements

1. **Zero-Copy Writes in `ObjectStoreWriter`**:
- `FileWrite::write` receives owned `bytes::Bytes`.
- `object_store::WriteMultipart` provides a `put(Bytes)` method that buffers without slice copying, avoiding redundant `extend_from_slice` allocations.

2. **S3 URL Boundary Validation**:
- Empty bucket strings (e.g. `s3:///path/to/file` or `s3://`) currently pass initial URL host parsing as `""`. These should explicitly return `ErrorKind::DataInvalid`.
- Ensure paths without trailing slashes (`s3://bucket`) extract `bucket` and empty relative path cleanly.

3. **Stream Handling & State Machine Guards**:
- Validate `delete_stream` on empty streams (`futures::stream::empty()`) and ensure immediate error propagation on invalid URLs.
- Add unit test coverage for `ObjectStoreWriter` state guards (e.g., verifying `close()` and `write()` error handling after writer has been closed).

4. **Backpressure & Memory Throttling**:
- For high-throughput streaming, evaluate `WriteMultipart::wait_for_capacity` to throttle producers and prevent unbounded 5MB chunk allocations in memory ahead of S3 network uploads.

### Related
- #2258 (Umbrella: Implement Storage using object_store)
- #2259 (Implement ObjectStoreStorage::S3)
- #3165 (feat(storage): Implement ObjectStoreStorage::S3)

## AI Disclosure
Drafted with AI assistance; reviewed and verified by contributor.

Contributor guide

Open the contributing guide

Research direction

Start with ObjectStoreStorage::S3, ObjectStoreWriter, FileWrite::write, and the WriteMultipart calls described in the issue. Check URL parsing for empty buckets and bucket-only paths, then inspect delete_stream and writer state handling before evaluating wait_for_capacity. Done means the listed edge cases return the expected errors, writes avoid redundant buffering, state-guard tests cover close/write failures, and streaming memory is appropriately bounded.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.