apache / apache/arrow-rs-object-store

Retrying failed part PUTs in multipart writes

Open
#831 4 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

SlateDB currently has a `RetryingObjectStore` that handles `ObjectStore` retries. The `object_store` retry mechanisms are unsuitable for us. This is something we've discussed in various GH issues in the past. We do something like this:

```rs
#[inline]
fn should_retry(err: &object_store::Error) -> bool {
let retry = !matches!(
err,
object_store::Error::AlreadyExists { .. }
| object_store::Error::Precondition { .. }
| object_store::Error::NotModified { .. }
| object_store::Error::NotFound { .. }
| object_store::Error::NotImplemented { .. }
| object_store::Error::NotSupported { .. }
);
if !retry {
debug!("not retrying object store operation [error={:?}]", err);
}
retry
}
```

This approach is somewhat similar to what Lance does (https://github.com/apache/arrow-rs-object-store/issues/737).

The problem we have right now is there's no good way for us to handle retries for multipart PUTs. We currently inherit `object_store`'s behavior for those writes. This behavior is insufficient (https://github.com/slatedb/slatedb/issues/1956).

**Describe the solution you'd like**

We've been wrestling a bit with the best way to handle this. The current `object_store` API makes it impossible for us to retry a multipart upload without buffering the entire byte object and retrying from scratch (as far as I can tell--please correct me if I'm wrong). I am seeking advice/suggestions at this point.

There are a few challenges here:

- The multipart puts are async. We don't see a part failure until later.
- We don't know which parts failed AFAICT.
- There's no high-level API for us to retry a failed part. I see `MultipartStore::put_part(.., part_idx, data)`, but that isn't exposed at the `ObjectStore` level.

What do you think? Should `object_store` to support custom retry policies for multipart uploads (or for uploads in general)? Should users be allowed to retry a specific part write?

It's quite possible I'm misunderstanding something here, so LMK if that's the case.

**Describe alternatives you've considered**

Living with what we have, which can halt SlateDB databases in certain scenarios.

**Additional context**

See GH links above.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing RetryingObjectStore and the MultipartStore::put_part API, then read the referenced SlateDB issue 1956 and object_store issue 737. The issue needs a decided approach for retrying failed multipart parts, including failure identification and retry policy; done would be an agreed API or implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.