fsspec / fsspec/s3fs

XAmzContentSHA256Mismatch writing to storage bucket

Open
#932 2 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

When writing to a Hetzner Storage Bucket, which supports same AWS and boto clients, I received an XAmzContentSHA256Mismatch

import s3fs

bucket_client = s3fs.S3FileSystem(
    key="***",
    secret="***",
    endpoint_url="**",
)

with bucket_client.open("s3://mybucker/test.txt", "wb") as f:
    f.write(b"eebhdbedbehdbehdbehj")

Logs:

File [~/miniconda/lib/python3.12/site-packages/s3fs/core.py:2290](http://localhost:8888/lab/workspaces/auto-P/tree/~/miniconda/lib/python3.12/site-packages/s3fs/core.py#line=2289), in S3File._call_s3(self, method, *kwarglist, **kwargs)
   2289 def _call_s3(self, method, *kwarglist, **kwargs):
-> 2290     return self.fs.call_s3(method, self.s3_additional_kwargs, *kwarglist, **kwargs)

File [~/miniconda/lib/python3.12/site-packages/fsspec/asyn.py:118](http://localhost:8888/lab/workspaces/auto-P/tree/~/miniconda/lib/python3.12/site-packages/fsspec/asyn.py#line=117), in sync_wrapper.<locals>.wrapper(*args, **kwargs)
    115 @functools.wraps(func)
    116 def wrapper(*args, **kwargs):
    117     self = obj or args[0]
--> 118     return sync(self.loop, func, *args, **kwargs)

File [~/miniconda/lib/python3.12/site-packages/fsspec/asyn.py:103](http://localhost:8888/lab/workspaces/auto-P/tree/~/miniconda/lib/python3.12/site-packages/fsspec/asyn.py#line=102), in sync(loop, func, timeout, *args, **kwargs)
    101     raise FSTimeoutError from return_result
    102 elif isinstance(return_result, BaseException):
--> 103     raise return_result
    104 else:
    105     return return_result

File [~/miniconda/lib/python3.12/site-packages/fsspec/asyn.py:56](http://localhost:8888/lab/workspaces/auto-P/tree/~/miniconda/lib/python3.12/site-packages/fsspec/asyn.py#line=55), in _runner(event, coro, result, timeout)
     54     coro = asyncio.wait_for(coro, timeout=timeout)
     55 try:
---> 56     result[0] = await coro
     57 except Exception as ex:
     58     result[0] = ex

File [~/miniconda/lib/python3.12/site-packages/s3fs/core.py:371](http://localhost:8888/lab/workspaces/auto-P/tree/~/miniconda/lib/python3.12/site-packages/s3fs/core.py#line=370), in S3FileSystem._call_s3(self, method, *akwarglist, **kwargs)
    369 logger.debug("CALL: %s - %s - %s", method.__name__, akwarglist, kw2)
    370 additional_kwargs = self._get_s3_method_kwargs(method, *akwarglist, **kwargs)
--> 371 return await _error_wrapper(
    372     method, kwargs=additional_kwargs, retries=self.retries
    373 )

File [~/miniconda/lib/python3.12/site-packages/s3fs/core.py:146](http://localhost:8888/lab/workspaces/auto-P/tree/~/miniconda/lib/python3.12/site-packages/s3fs/core.py#line=145), in _error_wrapper(func, args, kwargs, retries)
    144         err = e
    145 err = translate_boto_error(err)
--> 146 raise err

OSError: [Errno 5] An error occurred (XAmzContentSHA256Mismatch) when calling the PutObject operation: None

Versions:

  • Python 3.12.2
  • s3fs==2024.12.0
  • aioboto3==13.4.0
  • aiobotocore==2.18.0
  • boto==2.49.0
  • boto3==1.36.1
  • botocore==1.36.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the failing PutObject path in s3fs/core.py around lines 371 and 2290, using the provided versions and endpoint. Trace through fsspec/asyn.py around lines 56-118 and reproduce the write; done is a confirmed write without XAmzContentSHA256Mismatch or a documented compatibility limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.