fsspec / fsspec/s3fs

Checksum failing with Multipart Uploads

Open
#672 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

When sending multi-part uploads with a S3 Integrity Checksum, it fails with an error indicating not all parts have the checksum enabled.

I was able to enable ChecksumAlgorithm by adding a s3_additional_kwargs to the S3FileSystem initialisation. ex:

s3 = s3fs.S3FileSystem(key = access_key, secret = secret_key, s3_additional_kwargs = {"ChecksumAlgorithm": "SHA256"})

When sending a larger file using Multi-Part Uploads, it yields the following trace.

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/s3fs/core.py", line 112, in _error_wrapper
    return await func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/aiobotocore/client.py", line 358, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CompleteMultipartUpload operation: The upload was created using a sha256 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/....", line 97, in execute
    fobj_construct2parquet(incoming, content_id, filesystem=s3fs)
  File "/opt/.....", line 67, in fobj_construct2parquet
    pq.write_table(table, fobj, filesystem=filesystem)
  File "/home/airflow/.local/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 2941, in write_table
    writer.write_table(table, row_group_size=row_group_size)
  File "/home/airflow/.local/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 956, in __exit__
    self.close()
  File "/home/airflow/.local/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 1029, in close
    self.file_handle.close()
  File "pyarrow/io.pxi", line 180, in pyarrow.lib.NativeFile.close
  File "/home/airflow/.local/lib/python3.9/site-packages/fsspec/spec.py", line 1741, in close
    self.flush(force=True)
  File "/home/airflow/.local/lib/python3.9/site-packages/fsspec/spec.py", line 1612, in flush
    if self._upload_chunk(final=force) is not False:
  File "/home/airflow/.local/lib/python3.9/site-packages/s3fs/core.py", line 2173, in _upload_chunk
    self.commit()
  File "/home/airflow/.local/lib/python3.9/site-packages/s3fs/core.py", line 2201, in commit
    write_result = self._call_s3(
  File "/home/airflow/.local/lib/python3.9/site-packages/s3fs/core.py", line 2040, in _call_s3
    return self.fs.call_s3(method, self.s3_additional_kwargs, *kwarglist, **kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/fsspec/asyn.py", line 113, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/fsspec/asyn.py", line 98, in sync
    raise return_result
  File "/home/airflow/.local/lib/python3.9/site-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/home/airflow/.local/lib/python3.9/site-packages/s3fs/core.py", line 339, in _call_s3
    return await _error_wrapper(
  File "/home/airflow/.local/lib/python3.9/site-packages/s3fs/core.py", line 139, in _error_wrapper
    raise err
OSError: [Errno 22] The upload was created using a sha256 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in s3fs/core.py at _upload_chunk, commit, and _call_s3, which appear in the traceback. Trace how multipart parts and the final CompleteMultipartUpload request are built when s3_additional_kwargs contains ChecksumAlgorithm. Reproduce the reported upload and verify that checksum-enabled multipart uploads complete successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.