ceph / ceph/s3-tests

test_object_checksum_sha256 fails on AWS S3 because it expects BadDigest when AWS returns InvalidRequest

Open Beginner friendly
#644 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
372
Forks
328
PR merge metrics
No merged PRs in 30d

Description

Test test_object_checksum_sha256() in s3tests_boto3/functional/test_s3.py has this test expecting an invalid checksum to respond with `BadDigest` but AWS S3 responds with `InvalidRequest` as seen below.

IMO AWS should respond with `InvalidDigest` since `bad` is not a base64 encoded value but it is what it is.

PS: any way to enable more logging, to see the request details and response bodies?

```
e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, Body=body, ChecksumAlgorithm='SHA256', ChecksumSHA256='bad')
status, error_code = _get_status_and_error_code(e.response)
assert status == 400
assert error_code == 'BadDigest'
```

```
% S3TEST_CONF=awss3tests.conf tox -- s3tests_boto3/functional/test_s3.py::test_object_checksum_sha256
S3TEST_CONF=awss3tests.conf tox -- s3tests_boto3/functional/test_s3.py::test_object_checksum_sha256
.pkg: _optional_hooks> python /opt/homebrew/Cellar/tox/4.25.0/libexec/lib/python3.13/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /opt/homebrew/Cellar/tox/4.25.0/libexec/lib/python3.13/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /opt/homebrew/Cellar/tox/4.25.0/libexec/lib/python3.13/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /opt/homebrew/Cellar/tox/4.25.0/libexec/lib/python3.13/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /opt/homebrew/Cellar/tox/4.25.0/libexec/lib/python3.13/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py: install_package> python -I -m pip install --force-reinstall --no-deps /Users/jamshid/work/github/s3-tests/.tox/.tmp/package/14/s3tests-0.0.1.tar.gz

py: commands[0]> pytest s3tests_boto3/functional/test_s3.py::test_object_checksum_sha256
============================================================ test session starts ============================================================
platform darwin -- Python 3.13.3, pytest-8.3.5, pluggy-1.5.0
cachedir: .tox/py/.pytest_cache
rootdir: /Users/jamshid/work/github/s3-tests
configfile: pytest.ini
collected 1 item

s3tests_boto3/functional/test_s3.py FE [100%]

================================================================== ERRORS ===================================================================
_____________________________________________ ERROR at teardown of test_object_checksum_sha256 ______________________________________________

configfile =

@pytest.fixture(autouse=True)
def setup_teardown(configfile):
setup()
yield
> teardown()

s3tests_boto3/functional/__init__.py:339:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
s3tests_boto3/functional/__init__.py:308: in teardown
nuke_prefixed_buckets(prefix=prefix)
s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets
raise err
s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets
nuke_bucket(client, bucket_name)
s3tests_boto3/functional/__init__.py:104: in nuke_bucket
delete = client.delete_objects(Bucket=bucket,
.tox/py/lib/python3.13/site-packages/botocore/client.py:570: in _api_call
return self._make_api_call(operation_name, kwargs)
.tox/py/lib/python3.13/site-packages/botocore/context.py:124: in wrapper
return func(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = , operation_name = 'DeleteObjects'
api_params = {'Bucket': 'cephtests-v36rbvww4hnsrt6uugh-1', 'BypassGovernanceRetention': True, 'Delete': {'Objects': [{'Key': 'myobj', 'VersionId': 'null'}], 'Quiet': True}}

@with_current_context()
def _make_api_call(self, operation_name, api_params):
operation_model = self._service_model.operation_model(operation_name)
service_name = self._service_model.service_name
history_recorder.record(
'API_CALL',
{
'service': service_name,
'operation': operation_name,
'params': api_params,
},
)
if operation_model.deprecated:
logger.debug(
'Warning: %s.%s() is deprecated', service_name, operation_name
)
request_context = {
'client_region': self.meta.region_name,
'client_config': self.meta.config,
'has_streaming_input': operation_model.has_streaming_input,
'auth_type': operation_model.resolved_auth_type,
'unsigned_payload': operation_model.unsigned_payload,
}

api_params = self._emit_api_params(
api_params=api_params,
operation_model=operation_model,
context=request_context,
)
(
endpoint_url,
additional_headers,
properties,
) = self._resolve_endpoint_ruleset(
operation_model, api_params, request_context
)
if properties:
# Pass arbitrary endpoint info with the Request
# for use during construction.
request_context['endpoint_properties'] = properties
request_dict = self._convert_to_request_dict(
api_params=api_params,
operation_model=operation_model,
endpoint_url=endpoint_url,
context=request_context,
headers=additional_headers,
)
resolve_checksum_context(request_dict, operation_model, api_params)

service_id = self._service_model.service_id.hyphenize()
handler, event_response = self.meta.events.emit_until_response(
f'before-call.{service_id}.{operation_name}',
model=operation_model,
params=request_dict,
request_signer=self._request_signer,
context=request_context,
)

if event_response is not None:
http, parsed_response = event_response
else:
maybe_compress_request(
self.meta.config, request_dict, operation_model
)
apply_request_checksum(request_dict)
http, parsed_response = self._make_request(
operation_model, request_dict, request_context
)

self.meta.events.emit(
f'after-call.{service_id}.{operation_name}',
http_response=http,
parsed=parsed_response,
model=operation_model,
context=request_context,
)

if http.status_code >= 300:
error_info = parsed_response.get("Error", {})
error_code = error_info.get("QueryErrorCode") or error_info.get(
"Code"
)
error_class = self.exceptions.from_code(error_code)
> raise error_class(parsed_response, operation_name)
E botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the DeleteObjects operation: x-amz-bypass-governance-retention is only applicable to Object Lock enabled buckets.

.tox/py/lib/python3.13/site-packages/botocore/client.py:1031: ClientError
----------------------------------------------------------- Captured stdout setup -----------------------------------------------------------
Done with cleanup of buckets in tests.
Done with cleanup of buckets in tests.
Done with cleanup of buckets in tests.
================================================================= FAILURES ==================================================================
________________________________________________________ test_object_checksum_sha256 ________________________________________________________

@pytest.mark.checksum
def test_object_checksum_sha256():
bucket = get_new_bucket()
client = get_client()

key = "myobj"
size = 1024
body = FakeWriteFile(size, 'A')
sha256sum = 'arcu6553sHVAiX4MjW0j7I7vD4w6R+Gz9Ok0Q9lTa+0='
response = client.put_object(Bucket=bucket, Key=key, Body=body, ChecksumAlgorithm='SHA256', ChecksumSHA256=sha256sum)
assert sha256sum == response['ChecksumSHA256']

response = client.head_object(Bucket=bucket, Key=key)
assert 'ChecksumSHA256' not in response
response = client.head_object(Bucket=bucket, Key=key, ChecksumMode='ENABLED')
assert sha256sum == response['ChecksumSHA256']

e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, Body=body, ChecksumAlgorithm='SHA256', ChecksumSHA256='bad')
status, error_code = _get_status_and_error_code(e.response)
assert status == 400
> assert error_code == 'BadDigest'
E AssertionError: assert 'InvalidRequest' == 'BadDigest'
E
E - BadDigest
E + InvalidRequest

s3tests_boto3/functional/test_s3.py:14398: AssertionError
----------------------------------------------------------- Captured stdout setup -----------------------------------------------------------
Done with cleanup of buckets in tests.
Done with cleanup of buckets in tests.
Done with cleanup of buckets in tests.
============================================================= warnings summary ==============================================================
s3tests_boto3/functional/test_s3.py: 11 warnings
/Users/jamshid/work/github/s3-tests/.tox/py/lib/python3.13/site-packages/botocore/auth.py:425: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
datetime_now = datetime.datetime.utcnow()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================== short test summary info ==========================================================
FAILED s3tests_boto3/functional/test_s3.py::test_object_checksum_sha256 - AssertionError: assert 'InvalidRequest' == 'BadDigest'
ERROR s3tests_boto3/functional/test_s3.py::test_object_checksum_sha256 - botocore.exceptions.ClientError: An error occurred (InvalidArgume...
================================================== 1 failed, 11 warnings, 1 error in 4.43s ==================================================
py: exit 1 (5.70 seconds) /Users/jamshid/work/github/s3-tests> pytest s3tests_boto3/functional/test_s3.py::test_object_checksum_sha256 pid=54661
py: FAIL code 1 (23.94=setup[18.24]+cmd[5.70] seconds)
evaluation failed :( (24.20 seconds)```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with test_object_checksum_sha256 in s3tests_boto3/functional/test_s3.py and reproduce it using the tox command shown in the issue. Check the AWS response code and the teardown failure in s3tests_boto3/functional/__init__.py, then update the test expectation as appropriate and rerun the targeted test to verify it passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.