apache / apache/arrow-rs-object-store

Enable AWS client to copy objects >5GB in size

Open
#563 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'm trying to copy objects within a bucket as part of my data ingestion workflow. However, when thse objects exceed 5GB in size, this fails with an error. There's a built-in limit in the S3 API for the maximum size of a copy operation.

**Describe the solution you'd like**

I've outlined the solution i'd prefer in #561 - essentially the copy method should detect large copies and transparently switch to a multipart workflow.

**Describe alternatives you've considered**

Alternatively, we could also expose this API as part of the multipart-upload API.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing copy method and the multipart-upload API, then review the approach outlined in issue #561. The work is done when copies over the S3 5GB limit transparently use a multipart workflow, or the API is exposed through multipart upload as decided by the implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.