apache / apache/arrow-rs-object-store
Support for appends in S3 Express
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
About [a year ago](https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-express-one-zone-append-data-object/), AWS added ability to append data to objects in S3 Express / Directory Buckets: https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-objects-append.html
Append is atomic, but has the limit that up to 10000 'parts' can be appended before the file needs to be rewritten. Low-level, appending is implemented by using `x-amz-write-offset-bytes` header, which users need to set to the current size of the file.
We've built pretty efficient system using this functionality with official AWS S3 client, but it would be nice if we could do this using object_store somehow. Is it possible?
**Describe the solution you'd like**
I can see two options:
1. Ideally a top-level "append" operation could be re-introduced (I found it was removed in 0.8). The reasoning for this was that only in-memory and filesystem supported it at the time. Currently there's a bunch of object stores that support it: S3 (directory buckets), Azure, Ceph, Tencent, Alibaba, etc.
2. Use Attributes / Extensions to provide the header directly. Looking at the code I haven't found a good way to do that - would be grateful if you could point me to it!
**Describe alternatives you've considered**
Using a different client is the only thing that comes to mind.
**Additional context**
--
Contributor guide
Research direction
Start with the linked AWS S3 Express append documentation and trace the crate's existing Attributes/Extensions handling to determine how the write-offset header can be represented. Compare that path with the requested top-level append operation and existing support across object stores; done means an agreed API direction and working append support for the stated S3 Express use case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- backend-api-design, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100