apache / apache/arrow-rs-object-store
Retrying failed part PUTs in multipart writes
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
SlateDB currently has a `RetryingObjectStore` that handles `ObjectStore` retries. The `object_store` retry mechanisms are unsuitable for us. This is something we've discussed in various GH issues in the past. We do something like this:
```rs
#[inline]
fn should_retry(err: &object_store::Error) -> bool {
let retry = !matches!(
err,
object_store::Error::AlreadyExists { .. }
| object_store::Error::Precondition { .. }
| object_store::Error::NotModified { .. }
| object_store::Error::NotFound { .. }
| object_store::Error::NotImplemented { .. }
| object_store::Error::NotSupported { .. }
);
if !retry {
debug!("not retrying object store operation [error={:?}]", err);
}
retry
}
```
This approach is somewhat similar to what Lance does (https://github.com/apache/arrow-rs-object-store/issues/737).
The problem we have right now is there's no good way for us to handle retries for multipart PUTs. We currently inherit `object_store`'s behavior for those writes. This behavior is insufficient (https://github.com/slatedb/slatedb/issues/1956).
**Describe the solution you'd like**
We've been wrestling a bit with the best way to handle this. The current `object_store` API makes it impossible for us to retry a multipart upload without buffering the entire byte object and retrying from scratch (as far as I can tell--please correct me if I'm wrong). I am seeking advice/suggestions at this point.
There are a few challenges here:
- The multipart puts are async. We don't see a part failure until later.
- We don't know which parts failed AFAICT.
- There's no high-level API for us to retry a failed part. I see `MultipartStore::put_part(.., part_idx, data)`, but that isn't exposed at the `ObjectStore` level.
What do you think? Should `object_store` to support custom retry policies for multipart uploads (or for uploads in general)? Should users be allowed to retry a specific part write?
It's quite possible I'm misunderstanding something here, so LMK if that's the case.
**Describe alternatives you've considered**
Living with what we have, which can halt SlateDB databases in certain scenarios.
**Additional context**
See GH links above.
Contributor guide
Research direction
Start by reviewing RetryingObjectStore and the MultipartStore::put_part API, then read the referenced SlateDB issue 1956 and object_store issue 737. The issue needs a decided approach for retrying failed multipart parts, including failure identification and retry policy; done would be an agreed API or implementation scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100