GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil crashes when listing bucket contents when virtual addressing style fails
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
**What happened?**:
I configured gsutil to work with an on-premises S3 data source using a boto3 config (see below). The on-premises data source uses [path addressing and not virtual addressing](https://docs.aws.amazon.com/cli/latest/topic/s3-config.html#addressing-style). The configuration I detail below should allow gsutil to correctly access GCS buckets using the gs:// prefix with an [OAuth 2 User Account](https://cloud.google.com/storage/docs/gsutil/addlhelp/CredentialTypesSupportingVariousUseCases) and S3 on-premises buckets, such as provided by minio front-ending a data fabric, using the s3:// prefix. It fails if virtual addressing is not supported, which is a rare and difficult configuration for on-premises S3 data stores.
I can list buckets using this command:
```
gsutil ls s3://
```
However, any attempt to list the contents of a bucket, using the `-r` flag, results in a gsutil crash, shown below:
```
$ gsutil ls -r s3://master.train/
Traceback (most recent call last):
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gsutil", line 21, in
gsutil.RunMain()
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gsutil.py", line 124, in RunMain
sys.exit(gslib.__main__.main())
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 433, in main
user_project=user_project)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 628, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 411, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/commands/ls.py", line 641, in RunCommand
listing_helper.ExpandUrlAndPrint(storage_url))
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/utils/ls_helper.py", line 369, in ExpandUrlAndPrint
print_initial_newline=False)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/utils/ls_helper.py", line 443, in _RecurseExpandUrlAndPrint
bucket_listing_fields=self.bucket_listing_fields):
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 498, in IterAll
expand_top_level_buckets=expand_top_level_buckets):
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 174, in __iter__
fields=bucket_listing_fields):
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/boto_translation.py", line 434, in ListObjects
for key in objects_iter:
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/bucketlistresultset.py", line 34, in bucket_lister
encoding_type=encoding_type)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/bucket.py", line 477, in get_all_keys
'', headers, **params)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/bucket.py", line 403, in _get_all
query_args=query_args)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/connection.py", line 684, in make_request
retry_handler=retry_handler
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/connection.py", line 1074, in make_request
retry_handler=retry_handler)
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/connection.py", line 1033, in _mexe
raise ex
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
```
**What you expected to happen?**:
I expected it to list the contents of the bucket.
Properly configured aws-cli or mc works correctly and falls back to path configuration when virtual configuration fails.
**How to reproduce it (as minimally and precisely as possible)?**:
The first two steps are [documented in the gcloud docs](https://cloud.google.com/storage/docs/gsutil/commands/config#description).
1. Install gsutil standalone or disable gcloud credential-passing to gsutil
```
gcloud config set pass_credentials_to_gsutil false
```
2. Set up gsutil configuration
```
gsutil config
```
3. This will create a `~/.boto` file. In that file, around line 42, there are settings for adding HMAC credentials for s3:// urls. Add the authentication tokens for your on-premises S3 data store and configure s3_host and s3_port. Here are example settings for a typical minio setup:
```
# To add HMAC aws credentials for "s3://" URIs, edit and uncomment the
# following two lines:
aws_access_key_id = minio
aws_secret_access_key = minio123
# The ability to specify an alternate storage host and port
# is primarily for cloud storage service developers.
# Setting a non-default gs_host only works if prefer_api=xml.
s3_host = minio
s3_port = 9000
# In some cases, (e.g. VPC requests) the "host" HTTP header should
# be different than the host used in the request URL.
#s3_host_header =
```
6. If you have not added SSL certificates, you'll need to add the following in the `[boto3]` section
```
is_secure = False
```
If configuration worked properly, if you do a
```
gsutil ls gs://
```
you should see a list of your buckets in gcs.
Doing a
```
gsutil ls s3://
```
Will show a list of the buckets in your minio server.
The crash occurs when attempting to list the contents of any of those buckets.
**Anything else we need to know?**:
Running the command with the -DD flag will clearly indicate it is trying to access a path-addressed bucket.
**Environment?**:
- gsutil version: 4.49
Contributor guide
Research direction
Start with gslib/boto_translation.py, gslib/vendored/boto/boto/s3/bucket.py, and gslib/vendored/boto/boto/s3/connection.py, following the traceback from ls.py through wildcard_iterator.py. Reproduce the issue with the documented ~/.boto settings and recursive s3:// listing. Done means gsutil lists the contents of a path-addressed bucket without crashing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100