Checksum failing with Multipart Uploads
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
When sending multi-part uploads with a S3 Integrity Checksum, it fails with an error indicating not all parts have the checksum enabled.
I was able to enable ChecksumAlgorithm by adding a s3_additional_kwargs to the S3FileSystem initialisation. ex:
s3 = s3fs.S3FileSystem(key = access_key, secret = secret_key, s3_additional_kwargs = {"ChecksumAlgorithm": "SHA256"})
When sending a larger file using Multi-Part Uploads, it yields the following trace.
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/s3fs/core.py", line 112, in _error_wrapper
return await func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/aiobotocore/client.py", line 358, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CompleteMultipartUpload operation: The upload was created using a sha256 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/....", line 97, in execute
fobj_construct2parquet(incoming, content_id, filesystem=s3fs)
File "/opt/.....", line 67, in fobj_construct2parquet
pq.write_table(table, fobj, filesystem=filesystem)
File "/home/airflow/.local/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 2941, in write_table
writer.write_table(table, row_group_size=row_group_size)
File "/home/airflow/.local/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 956, in __exit__
self.close()
File "/home/airflow/.local/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 1029, in close
self.file_handle.close()
File "pyarrow/io.pxi", line 180, in pyarrow.lib.NativeFile.close
File "/home/airflow/.local/lib/python3.9/site-packages/fsspec/spec.py", line 1741, in close
self.flush(force=True)
File "/home/airflow/.local/lib/python3.9/site-packages/fsspec/spec.py", line 1612, in flush
if self._upload_chunk(final=force) is not False:
File "/home/airflow/.local/lib/python3.9/site-packages/s3fs/core.py", line 2173, in _upload_chunk
self.commit()
File "/home/airflow/.local/lib/python3.9/site-packages/s3fs/core.py", line 2201, in commit
write_result = self._call_s3(
File "/home/airflow/.local/lib/python3.9/site-packages/s3fs/core.py", line 2040, in _call_s3
return self.fs.call_s3(method, self.s3_additional_kwargs, *kwarglist, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/fsspec/asyn.py", line 113, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/fsspec/asyn.py", line 98, in sync
raise return_result
File "/home/airflow/.local/lib/python3.9/site-packages/fsspec/asyn.py", line 53, in _runner
result[0] = await coro
File "/home/airflow/.local/lib/python3.9/site-packages/s3fs/core.py", line 339, in _call_s3
return await _error_wrapper(
File "/home/airflow/.local/lib/python3.9/site-packages/s3fs/core.py", line 139, in _error_wrapper
raise err
OSError: [Errno 22] The upload was created using a sha256 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in s3fs/core.py at _upload_chunk, commit, and _call_s3, which appear in the traceback. Trace how multipart parts and the final CompleteMultipartUpload request are built when s3_additional_kwargs contains ChecksumAlgorithm. Reproduce the reported upload and verify that checksum-enabled multipart uploads complete successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100