merge files fails with small or large files
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
Sorry if there is already an issue for this. the word "merge" is difficult to search for due to its use in git.
The merge function throws an error if any parts are outside the valid part size range for an s3 multipart upload.
Large input files can become multiple parts using the CopySourceRange parameter of upload_part_copy
Smaller parts will either need at least 5MiB of data to be downloaded or temporary files on s3.
Here are some failing tests
def test_merge_small_part(s3):
with s3.open(a, "wb") as f:
f.write(b"a" * 4 * 2**20)
with s3.open(b, "wb") as f:
f.write(b"a" * 10 * 2**20)
s3.merge(test_bucket_name + "/joined", [a, b])
assert s3.info(test_bucket_name + "/joined")["size"] == (4 * 2**20) + (10 * 2**20)
def test_merge_large_part(s3):
with s3.open(a, "wb") as f:
f.write(b"a" * 6 * 2**30)
with s3.open(b, "wb") as f:
f.write(b"a" * 10 * 2**20)
s3.merge(test_bucket_name + "/joined", [a, b])
assert s3.info(test_bucket_name + "/joined")["size"] == (6 * 2**30) + (10 * 2**20)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the s3.merge entry point and inspect how it uses upload_part_copy and the CopySourceRange parameter. Reproduce the two failing cases shown in the issue, covering both a small and a large part. Done means both tests complete successfully and the joined object size matches the sum of the inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100