apache / apache/libcloud

MinIO: ObjectHashMismatchError when trying to upload object

Open
#1,607 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.1k
Forks
931
Avg merge
1d 2h
Merged PRs (30d)
4

Description

## Summary

The MinIO provider raises an `ObjectHashMismatchError` when trying to upload an object, for no clear reason
100% reproducible, code to repro the bug is provided

## Detailed Information

libcloud: latest stable, 3.3.1
Python: 3.8.10
OS: Ubuntu 20.04

Here you can find a repo to repro the bug, using `pytest` (Actually there are 2 bugs to report):
https://github.com/Wenzel/libcloud_bug

~~~python3
$ pytest -k test_demo_ObjectHashMismatchError_with_pyfakefs
clean_minio_db = , fs =

def test_demo_ObjectHashMismatchError_with_pyfakefs(clean_minio_db, fs):
# create test file
test_file = "/file1.txt"
test_file_data = b"hello"
with open(test_file, "wb") as f:
f.write(test_file_data)
# create container
driver = clean_minio_db
container = driver.create_container('test')
# test
> driver.upload_object(str(test_file), container, 'test_file')

/home/wenzel/Projets/libcloud_bug/tests/test_demo_bug.py:13:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/wenzel/.cache/pypoetry/virtualenvs/libcloud-bug-IT-EyGKG-py3.8/lib/python3.8/site-packages/libcloud/storage/drivers/s3.py:545: in upload_object
return self._put_object(container=container, object_name=object_name,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = , container = , object_name = 'test_file', method = 'PUT'
query_args = None, extra = {}, file_path = '/file1.txt', stream = None, verify_hash = True, storage_class = None, headers = {'connection': 'close', 'content-type': 'text/plain; charset=utf-8'}

def _put_object(self, container, object_name, method='PUT',
query_args=None, extra=None, file_path=None,
stream=None, verify_hash=True, storage_class=None,
headers=None):
headers = headers or {}
extra = extra or {}

headers.update(self._to_storage_class_headers(storage_class))

content_type = extra.get('content_type', None)
meta_data = extra.get('meta_data', None)
acl = extra.get('acl', None)

if meta_data:
for key, value in list(meta_data.items()):
key = self.http_vendor_prefix + '-meta-%s' % (key)
headers[key] = value

if acl:
headers[self.http_vendor_prefix + '-acl'] = acl

request_path = self._get_object_path(container, object_name)

if query_args:
request_path = '?'.join((request_path, query_args))

result_dict = self._upload_object(
object_name=object_name, content_type=content_type,
request_path=request_path, request_method=method,
headers=headers, file_path=file_path, stream=stream)

response = result_dict['response']
bytes_transferred = result_dict['bytes_transferred']
headers = response.headers
response = response
server_hash = headers.get('etag', '').replace('"', '')
server_side_encryption = headers.get('x-amz-server-side-encryption',
None)
aws_kms_encryption = (server_side_encryption == 'aws:kms')
hash_matches = (result_dict['data_hash'] == server_hash)

# NOTE: If AWS KMS server side encryption is enabled, ETag won't
# contain object MD5 digest so we skip the checksum check
# See https://docs.aws.amazon.com/AmazonS3/latest/API
# /RESTCommonResponseHeaders.html
# and https://github.com/apache/libcloud/issues/1401
# for details
if verify_hash and not aws_kms_encryption and not hash_matches:
> raise ObjectHashMismatchError(
value='MD5 hash {0} checksum does not match {1}'.format(
server_hash, result_dict['data_hash']),
E libcloud.storage.types.ObjectHashMismatchError: , value=MD5 hash checksum does not match 5d41402abc4b2a76b9719d911017c592, object = test_file>

/home/wenzel/.cache/pypoetry/virtualenvs/libcloud-bug-IT-EyGKG-py3.8/lib/python3.8/site-packages/libcloud/storage/drivers/s3.py:922: ObjectHashMismatchError
------------------------------------------------------------------------------------------- Captured stdout setup -------------------------------------------------------------------------------------------
db2dfd078b96e1b0932dc354926aadb77adba2d8c19b8ccc875e7fe5163e8f46
----------------------------------------------------------------------------------------- Captured stdout teardown ------------------------------------------------------------------------------------------
libcloud_bug_objectmistmatch_miniodb
========================================================================================== short test summary info ==========================================================================================
FAILED tests/test_demo_bug.py::test_demo_ObjectHashMismatchError_with_pyfakefs - libcloud.storage.types.ObjectHashMismatchError:

Contributor guide

Open the contributing guide

Research direction

Start with the MinIO provider and the S3 upload path shown in libcloud/storage/drivers/s3.py, especially _put_object and the checksum comparison around line 922. Run the linked pytest reproduction, test_demo_ObjectHashMismatchError_with_pyfakefs, and compare the upload response headers with the calculated hash. Done means the reproducible upload succeeds without an incorrect ObjectHashMismatchError and relevant tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.