ansible-collections / ansible-collections/google.cloud

google.cloud.gcp_storage_object module unable to download gkeadm from public bucket

Open
#575 4 comments 1 reaction 0 assignees View on GitHub
stale
Dominant language
Python
Stars
105
Forks
144
Avg merge
5d 10h
Merged PRs (30d)
4

Description

##### SUMMARY
Unable to download "gkeadm" binary from gke-on-prem-release bucket using google.cloud.gcp_storage_object module using a service account. Always given 403 error. manual download with a serviceaccount and using gsutil works.
##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME

google.cloud.gcp_storage_object

##### ANSIBLE VERSION

```paste below
ansible [core 2.13.9]
config file = None
configured module search path = ['/home/grsiepka/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/ansible
ansible collection location = /home/grsiepka/.ansible/collections:/usr/share/ansible/collections
executable location = /home/grsiepka/github/gene-test/ansible/.venv/bin/ansible
python version = 3.8.10 (default, Mar 13 2023, 10:26:41) [GCC 9.4.0]
jinja version = 3.1.2
libyaml = True

```

##### COLLECTION VERSION

```paste below
Collection Version
------------ -------
google.cloud 1.1.3
```

##### CONFIGURATION

```paste below
```

##### OS / ENVIRONMENT

Ubuntu 20.04 in WSL under python VirtualENV

##### STEPS TO REPRODUCE

```yaml
- name: Setup
hosts: localhost
gather_facts: true
tasks:
- name: Grab gkeadm binary for our version
google.cloud.gcp_storage_object:
action: download
bucket: gke-on-prem-release
src: gkeadm/1.15.0-gke.581/linux/gkeadm
dest: gkeadm
auth_kind: serviceaccount
service_account_file: mygcpsakeyfile.json

```

##### EXPECTED RESULTS

File downloaded

##### ACTUAL RESULTS

File not downloaded, 403 error Output below modified to hide GCP SA email "MYSERVICEACCOUNT@email"

```paste below
TASK [Grab gkeadm binary for our version] **********************************************************************************************************
task path: /home/grsiepka/github/gene-test/ansible/playbooks/test.yml:6
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: grsiepka
<127.0.0.1> EXEC /bin/sh -c 'echo ~grsiepka && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/grsiepka/.ansible/tmp `"&& mkdir "` echo /home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457 `" && echo ansible-tmp-1686062920.8065732-11369-112001459093457="` echo /home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457 `" ) && sleep 0'
Using module file /home/grsiepka/.ansible/collections/ansible_collections/google/cloud/plugins/modules/gcp_storage_object.py
<127.0.0.1> PUT /home/grsiepka/.ansible/tmp/ansible-local-11244t356t8j6/tmp8zo_1mv5 TO /home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457/AnsiballZ_gcp_storage_object.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457/ /home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457/AnsiballZ_gcp_storage_object.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/home/grsiepka/github/gene-test/ansible/.venv/bin/python3 /home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457/AnsiballZ_gcp_storage_object.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
File "/home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457/AnsiballZ_gcp_storage_object.py", line 107, in
_ansiballz_main()
File "/home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457/AnsiballZ_gcp_storage_object.py", line 99, in _ansiballz_main
invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
File "/home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457/AnsiballZ_gcp_storage_object.py", line 47, in invoke_module
runpy.run_module(mod_name='ansible_collections.google.cloud.plugins.modules.gcp_storage_object', init_globals=dict(_module_fqn='ansible_collections.google.cloud.plugins.modules.gcp_storage_object', _modlib_path=modlib_path),
File "/usr/lib/python3.8/runpy.py", line 207, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/tmp/ansible_google.cloud.gcp_storage_object_payload_7oa3pine/ansible_google.cloud.gcp_storage_object_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_storage_object.py", line 305, in
File "/tmp/ansible_google.cloud.gcp_storage_object_payload_7oa3pine/ansible_google.cloud.gcp_storage_object_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_storage_object.py", line 207, in main
File "/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/cloud/storage/client.py", line 773, in get_bucket
bucket.reload(
File "/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/cloud/storage/bucket.py", line 1077, in reload
super(Bucket, self).reload(
File "/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/cloud/storage/_helpers.py", line 246, in reload
api_response = client._get_resource(
File "/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/cloud/storage/client.py", line 378, in _get_resource
return self._connection.api_request(
File "/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/cloud/storage/_http.py", line 72, in api_request
return call()
File "/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
return retry_target(
File "/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/api_core/retry.py", line 191, in retry_target
return target()
File "/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/cloud/_http/__init__.py", line 494, in api_request
raise exceptions.from_http_response(response)
google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/gke-on-prem-release?projection=noAcl&prettyPrint=false: MYSERVICEACCOUNT@email does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist).
fatal: [localhost]: FAILED! => {
"changed": false,
"module_stderr": "Traceback (most recent call last):\n File \"/home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457/AnsiballZ_gcp_storage_object.py\", line 107, in \n _ansiballz_main()\n File \"/home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457/AnsiballZ_gcp_storage_object.py\", line 99, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/home/grsiepka/.ansible/tmp/ansible-tmp-1686062920.8065732-11369-112001459093457/AnsiballZ_gcp_storage_object.py\", line 47, in invoke_module\n runpy.run_module(mod_name='ansible_collections.google.cloud.plugins.modules.gcp_storage_object', init_globals=dict(_module_fqn='ansible_collections.google.cloud.plugins.modules.gcp_storage_object', _modlib_path=modlib_path),\n File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_google.cloud.gcp_storage_object_payload_7oa3pine/ansible_google.cloud.gcp_storage_object_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_storage_object.py\", line 305, in \n File \"/tmp/ansible_google.cloud.gcp_storage_object_payload_7oa3pine/ansible_google.cloud.gcp_storage_object_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_storage_object.py\", line 207, in main\n File \"/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/cloud/storage/client.py\", line 773, in get_bucket\n bucket.reload(\n File \"/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/cloud/storage/bucket.py\", line 1077, in reload\n super(Bucket, self).reload(\n File \"/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/cloud/storage/_helpers.py\", line 246, in reload\n api_response = client._get_resource(\n File \"/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/cloud/storage/client.py\", line 378, in _get_resource\n return self._connection.api_request(\n File \"/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/cloud/storage/_http.py\", line 72, in api_request\n return call()\n File \"/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/api_core/retry.py\", line 349, in retry_wrapped_func\n return retry_target(\n File \"/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/api_core/retry.py\", line 191, in retry_target\n return target()\n File \"/home/grsiepka/github/gene-test/ansible/.venv/lib/python3.8/site-packages/google/cloud/_http/__init__.py\", line 494, in api_request\n raise exceptions.from_http_response(response)\ngoogle.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/gke-on-prem-release?projection=noAcl&prettyPrint=false: MYSERVICEACCOUNT@email does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist).\n",
"module_stdout": "",
"msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
"rc": 1
}

```

Contributor guide

Open the contributing guide

Research direction

Start with plugins/modules/gcp_storage_object.py, especially main and the google.cloud.storage client get_bucket call shown in the traceback. Run the playbook task from playbooks/test.yml with the service account, then compare its bucket access with the successful gsutil download. Done means the specified gkeadm object downloads without the reported 403 error.

Written by the indexing model from the issue text.

Assessment

Tech stack
ansible, google-cloud, python
Domain
cloud, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.