S3 Function Header tests seems faulty (executed on ceph vm cluster)

Open

#438 opened on Apr 5, 2022 1 comment 0 reactions 0 assignees View on GitHub

Dominant language: Python
Stars: 372
Forks: 328
PR merge metrics: No merged PRs in 30d

Description

**Faulty Tests:**
- test_bucket_create_bad_authorization_empty
- test_bucket_create_bad_authorization_none
- test_bucket_create_bad_authorization_invalid_aws2
- test_bucket_create_bad_date_none_aws2

**Issue Description:**
The above-mentioned tests are failing on ceph cluster deployed in the virtual machine environment.
When the same tests are performed using REST API, we get the appropriate response. (403 Forbidden)

Ceph/S3-tests Failure Output
The test is to set the Authorization header to "" (empty), from the debug request header log we can see that the Authorization header was not set to "".

```bash
$ S3TEST_CONF=./s3tests.motr.conf virtualenv/bin/nosetests s3tests_boto3.functional.test_headers:test_bucket_create_bad_authorization_empty -v 2>&1 | tee -a cephtest.log
s3tests_boto3.functional.test_headers.test_bucket_create_bad_authorization_empty ... FAIL

======================================================================
FAIL: s3tests_boto3.functional.test_headers.test_bucket_create_bad_authorization_empty
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/ceph-tests/s3-tests/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/root/ceph-tests/s3-tests/s3tests_boto3/functional/test_headers.py", line 500, in test_bucket_create_bad_authorization_empty
e = _add_header_create_bad_bucket(headers)
File "/root/ceph-tests/s3-tests/s3tests_boto3/functional/test_headers.py", line 113, in _add_header_create_bad_bucket
e = assert_raises(ClientError, client.create_bucket, Bucket=bucket_name)
File "/root/ceph-tests/s3-tests/s3tests_boto3/functional/utils.py", line 21, in assert_raises
raise AssertionError("%s not raised" % excName)
AssertionError: ClientError not raised
-------------------- >> begin captured logging << --------------------
botocore.hooks: DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
botocore.hooks: DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway
botocore.hooks: DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
botocore.hooks: DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
botocore.hooks: DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
botocore.hooks: DEBUG: Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
botocore.hooks: DEBUG: Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
botocore.hooks: DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
botocore.hooks: DEBUG: Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
botocore.hooks: DEBUG: Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
botocore.hooks: DEBUG: Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
botocore.loaders: DEBUG: Loading JSON file: /root/ceph-tests/s3-tests/virtualenv/lib/python3.6/site-packages/botocore/data/endpoints.json
botocore.loaders: DEBUG: Loading JSON file: /root/ceph-tests/s3-tests/virtualenv/lib/python3.6/site-packages/botocore/data/sdk-default-configuration.json
botocore.hooks: DEBUG: Event choose-service-name: calling handler
botocore.loaders: DEBUG: Loading JSON file: /root/ceph-tests/s3-tests/virtualenv/lib/python3.6/site-packages/botocore/data/s3/2006-03-01/service-2.json
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f007fe9e730>
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler
botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60)
botocore.loaders: DEBUG: Loading JSON file: /root/ceph-tests/s3-tests/virtualenv/lib/python3.6/site-packages/botocore/data/_retry.json
botocore.client: DEBUG: Registering retry handlers for service: s3
botocore.hooks: DEBUG: Event choose-service-name: calling handler
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f007fe9e730>
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler
botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60)
botocore.client: DEBUG: Registering retry handlers for service: s3
botocore.hooks: DEBUG: Event choose-service-name: calling handler
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f007fe9e730>
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler
botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60)
botocore.client: DEBUG: Registering retry handlers for service: s3
botocore.hooks: DEBUG: Event before-parameter-build.s3.ListBuckets: calling handler
botocore.hooks: DEBUG: Event before-parameter-build.s3.ListBuckets: calling handler >
botocore.hooks: DEBUG: Event before-parameter-build.s3.ListBuckets: calling handler >
botocore.hooks: DEBUG: Event before-parameter-build.s3.ListBuckets: calling handler
botocore.hooks: DEBUG: Event before-call.s3.ListBuckets: calling handler
botocore.hooks: DEBUG: Event before-call.s3.ListBuckets: calling handler >
botocore.hooks: DEBUG: Event before-call.s3.ListBuckets: calling handler
botocore.hooks: DEBUG: Event before-call.s3.ListBuckets: calling handler
botocore.endpoint: DEBUG: Making request for OperationModel(name=ListBuckets) with params: {'url_path': '/', 'query_string': '', 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.21.33 Python/3.6.8 Linux/4.18.0-305.3.1.el8_4.x86_64 Botocore/1.24.33'}, 'body': b'', 'url': 'http://s3.seagate.com:8000/', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': None}}}
botocore.hooks: DEBUG: Event request-created.s3.ListBuckets: calling handler >
botocore.hooks: DEBUG: Event choose-signer.s3.ListBuckets: calling handler >
botocore.hooks: DEBUG: Event choose-signer.s3.ListBuckets: calling handler
botocore.hooks: DEBUG: Event before-sign.s3.ListBuckets: calling handler >
botocore.utils: DEBUG: Using S3 path style addressing.
botocore.auth: DEBUG: Calculating signature using v4 auth.
botocore.auth: DEBUG: CanonicalRequest:
GET
/

host:s3.seagate.com:8000
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220405T112900Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
botocore.auth: DEBUG: StringToSign:
AWS4-HMAC-SHA256
20220405T112900Z
20220405/us-east-1/s3/aws4_request
22228fc1e8b134ddb0eb200e0ad20cba21136163ec54d59d8cce3de3cf9f3ece
botocore.auth: DEBUG: Signature:
3c4a33971d3a168f1df5024128387f34a21901337e6c0b9c400e725f6b538a06
botocore.hooks: DEBUG: Event request-created.s3.ListBuckets: calling handler
botocore.endpoint: DEBUG: Sending http request:
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): s3.seagate.com:8000
urllib3.connectionpool: DEBUG: http://s3.seagate.com:8000 "GET / HTTP/1.1" 200 None
botocore.parsers: DEBUG: Response headers: {'Transfer-Encoding': 'chunked', 'Content-Type': 'application/xml', 'Date': 'Tue, 05 Apr 2022 11:29:00 GMT', 'Connection': 'Keep-Alive'}
botocore.parsers: DEBUG: Response body:
b'mgwuserSuperUseremptybkt2022-03-07T08:43:27.486Zhehe2022-04-04T12:34:53.828Zmybkt2022-02-24T05:35:23.667Zmybkt22022-03-04T05:46:30.031Ztestbkt2022-02-24T11:47:30.855Ztestbkt12022-04-04T05:56:54.843Ztestbkt22022-02-24T12:16:52.286Z'
botocore.hooks: DEBUG: Event needs-retry.s3.ListBuckets: calling handler
botocore.retryhandler: DEBUG: No retry needed.
botocore.hooks: DEBUG: Event needs-retry.s3.ListBuckets: calling handler >
botocore.hooks: DEBUG: Event before-parameter-build.s3.ListBuckets: calling handler
botocore.hooks: DEBUG: Event before-parameter-build.s3.ListBuckets: calling handler >
botocore.hooks: DEBUG: Event before-parameter-build.s3.ListBuckets: calling handler >
botocore.hooks: DEBUG: Event before-parameter-build.s3.ListBuckets: calling handler
botocore.hooks: DEBUG: Event before-call.s3.ListBuckets: calling handler
botocore.hooks: DEBUG: Event before-call.s3.ListBuckets: calling handler >
botocore.hooks: DEBUG: Event before-call.s3.ListBuckets: calling handler
botocore.hooks: DEBUG: Event before-call.s3.ListBuckets: calling handler
botocore.endpoint: DEBUG: Making request for OperationModel(name=ListBuckets) with params: {'url_path': '/', 'query_string': '', 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.21.33 Python/3.6.8 Linux/4.18.0-305.3.1.el8_4.x86_64 Botocore/1.24.33'}, 'body': b'', 'url': 'http://s3.seagate.com:8000/', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': None}}}
botocore.hooks: DEBUG: Event request-created.s3.ListBuckets: calling handler >
botocore.hooks: DEBUG: Event choose-signer.s3.ListBuckets: calling handler >
botocore.hooks: DEBUG: Event choose-signer.s3.ListBuckets: calling handler
botocore.hooks: DEBUG: Event before-sign.s3.ListBuckets: calling handler >
botocore.utils: DEBUG: Using S3 path style addressing.
botocore.auth: DEBUG: Calculating signature using v4 auth.
botocore.auth: DEBUG: CanonicalRequest:
GET
/

host:s3.seagate.com:8000
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220405T112900Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
botocore.auth: DEBUG: StringToSign:
AWS4-HMAC-SHA256
20220405T112900Z
20220405/us-east-1/s3/aws4_request
22228fc1e8b134ddb0eb200e0ad20cba21136163ec54d59d8cce3de3cf9f3ece
botocore.auth: DEBUG: Signature:
3c4a33971d3a168f1df5024128387f34a21901337e6c0b9c400e725f6b538a06
botocore.hooks: DEBUG: Event request-created.s3.ListBuckets: calling handler
botocore.endpoint: DEBUG: Sending http request:
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): s3.seagate.com:8000
urllib3.connectionpool: DEBUG: http://s3.seagate.com:8000 "GET / HTTP/1.1" 200 None
botocore.parsers: DEBUG: Response headers: {'Transfer-Encoding': 'chunked', 'Content-Type': 'application/xml', 'Date': 'Tue, 05 Apr 2022 11:29:00 GMT', 'Connection': 'Keep-Alive'}
botocore.parsers: DEBUG: Response body:
b'mgwuserSuperUseremptybkt2022-03-07T08:43:27.486Zhehe2022-04-04T12:34:53.828Zmybkt2022-02-24T05:35:23.667Zmybkt22022-03-04T05:46:30.031Ztestbkt2022-02-24T11:47:30.855Ztestbkt12022-04-04T05:56:54.843Ztestbkt22022-02-24T12:16:52.286Z'
botocore.hooks: DEBUG: Event needs-retry.s3.ListBuckets: calling handler
botocore.retryhandler: DEBUG: No retry needed.
botocore.hooks: DEBUG: Event needs-retry.s3.ListBuckets: calling handler >
botocore.hooks: DEBUG: Event choose-service-name: calling handler
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f007fe9e730>
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler
botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60)
botocore.client: DEBUG: Registering retry handlers for service: s3
botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler
botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler >
botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler >
botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler
botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler . at 0x7f007f50f620>
botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler
botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler >
botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler
botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler
botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/sumedh-t1ssyn1xpcjl2mzh4mcq9e-1', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.21.33 Python/3.6.8 Linux/4.18.0-305.3.1.el8_4.x86_64 Botocore/1.24.33', 'Authorization': ''}, 'body': b'', 'url': 'http://s3.seagate.com:8000/sumedh-t1ssyn1xpcjl2mzh4mcq9e-1', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'sumedh-t1ssyn1xpcjl2mzh4mcq9e-1'}}}
botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler >
botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler >
botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler
botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler >
botocore.utils: DEBUG: Using S3 path style addressing.
botocore.auth: DEBUG: Calculating signature using v4 auth.
botocore.auth: DEBUG: CanonicalRequest:
PUT
/sumedh-t1ssyn1xpcjl2mzh4mcq9e-1

host:s3.seagate.com:8000
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220405T112900Z

----------------------------------------------------------------------
Ran 1 test in 0.796s

FAILED (failures=1)
(virtualenv)
```

Test output using REST API

```bash
~/ceph-client ⌚ 4:29:18
$ cd /root/ceph-client ; /usr/bin/env /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-2.el8_5.x86_64/bin/java -ag
entlib:jdwp=transport=dt_socket,server=n,suspend=y,address=localhost:34569 -cp /tmp/cp_bbidnu5w7per3qn9hi9pyfa63
.jar com.seagate.ceph_client.CephAdminAPIClient
Date: Mon, 4 Apr 2022 10:35:34 GMT
HeaderString: PUT

Mon, 4 Apr 2022 10:35:34 GMT
/hehe
Auth:

HTTP/1.1 403 ForbiddenResponse Content is: {"Code":"AccessDenied","BucketName":"hehe","RequestId":"tx000008d3648d298f720bc-00624ac9f7-1050-default","HostId":"1050-default-default"}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

S3 Function Header tests seems faulty (executed on ceph vm cluster)

Description

Contributor guide

Assessment

Get new issues in your inbox