fsspec / fsspec/s3fs

merge files fails with small or large files

Open
#778 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

Sorry if there is already an issue for this. the word "merge" is difficult to search for due to its use in git.

The merge function throws an error if any parts are outside the valid part size range for an s3 multipart upload.

Large input files can become multiple parts using the CopySourceRange parameter of upload_part_copy
Smaller parts will either need at least 5MiB of data to be downloaded or temporary files on s3.

Here are some failing tests

def test_merge_small_part(s3):
    with s3.open(a, "wb") as f:
        f.write(b"a" * 4 * 2**20)

    with s3.open(b, "wb") as f:
        f.write(b"a" * 10 * 2**20)
    s3.merge(test_bucket_name + "/joined", [a, b])
    assert s3.info(test_bucket_name + "/joined")["size"] == (4 * 2**20) + (10 * 2**20)

def test_merge_large_part(s3):
    with s3.open(a, "wb") as f:
        f.write(b"a" * 6 * 2**30)

    with s3.open(b, "wb") as f:
        f.write(b"a" * 10 * 2**20)
    s3.merge(test_bucket_name + "/joined", [a, b])
    assert s3.info(test_bucket_name + "/joined")["size"] == (6 * 2**30) + (10 * 2**20)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the s3.merge entry point and inspect how it uses upload_part_copy and the CopySourceRange parameter. Reproduce the two failing cases shown in the issue, covering both a small and a large part. Done means both tests complete successfully and the joined object size matches the sum of the inputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.