apache / apache/arrow-rs-object-store

Support for appends in S3 Express

Open
#632 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

About [a year ago](https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-express-one-zone-append-data-object/), AWS added ability to append data to objects in S3 Express / Directory Buckets: https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-objects-append.html

Append is atomic, but has the limit that up to 10000 'parts' can be appended before the file needs to be rewritten. Low-level, appending is implemented by using `x-amz-write-offset-bytes` header, which users need to set to the current size of the file.

We've built pretty efficient system using this functionality with official AWS S3 client, but it would be nice if we could do this using object_store somehow. Is it possible?

**Describe the solution you'd like**

I can see two options:
1. Ideally a top-level "append" operation could be re-introduced (I found it was removed in 0.8). The reasoning for this was that only in-memory and filesystem supported it at the time. Currently there's a bunch of object stores that support it: S3 (directory buckets), Azure, Ceph, Tencent, Alibaba, etc.
2. Use Attributes / Extensions to provide the header directly. Looking at the code I haven't found a good way to do that - would be grateful if you could point me to it!

**Describe alternatives you've considered**

Using a different client is the only thing that comes to mind.

**Additional context**

--

Contributor guide

Open the contributing guide

Research direction

Start with the linked AWS S3 Express append documentation and trace the crate's existing Attributes/Extensions handling to determine how the write-offset header can be represented. Compare that path with the requested top-level append operation and existing support across object stores; done means an agreed API direction and working append support for the stated S3 Express use case.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
backend-api-design, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.