gcp_storage_object Python3 UnicodeDecodeError: 'utf-8' codec can't decode

_From @bucklander on Jul 31, 2020 22:41_

##### SUMMARY
While using Python3, gcp_storage_object module is unable to upload binary files (and perhaps other file types) without throwing:
`UnicodeDecodeError: 'utf-8' codec can't decode byte 0x98 in position 522: invalid start byte`

This is possibly related to [Ansible python3's explicit string conversion](https://docs.ansible.com/ansible/2.3/dev_guide/developing_python3.html#background) (or not). When using other file types (E.g. simple text files) or just using Python2 with any file type, the problem does not exist.

Below are two example playbooks; One which uploads a gzip tarball file, and another which uploads a simple text file. Each playbook is executed with Python2 and again with Python3.

I'm certainly no expert in file compression or file types, however in my particular case the gzip'd file format seems to cause the issue. I'm also seeing this problem with binary files within regular tarball (no gzip compression). You could also replace the compressed file examples below with seemingly any randomly generated binary file, and the result appears to be the same.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
gcp_storage_object

##### ANSIBLE VERSION

```
$ ansible --version
ansible 2.9.11
config file = None
configured module search path = ['${HOME}/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible
executable location = /Library/Frameworks/Python.framework/Versions/3.8/bin/ansible
python version = 3.8.5 (v3.8.5:580fbb018f, Jul 20 2020, 12:11:27) [Clang 6.0 (clang-600.0.57)]
```

##### CONFIGURATION

```
$ ansible-config dump --only-changed
$
```

##### OS / ENVIRONMENT

This can be reproduced on a few different systems. Here are a few specific examples:
```
macOS Mojave 10.14.6
Python 2.7.15
Python 3.8.5

CentOS Linux release 8.1.1911 (Core)
Python 3.6.8
```

##### STEPS TO REPRODUCE

Execute the following example playbooks with a simple, flat text file as well as a tarball.
```
echo "hello" > foobar.txt
tar -zcf binary.tar foobar.txt
```

Example playbook to upload binary file to GCS:
```
# upload-bin-file.yml
---

- name: Upload a binary file to Google Cloud Storage bucket using gcp_storage_object module
hosts: all
connection: local
tasks:
- name: Upload Latest Backup(s) to GCS Bucket
gcp_storage_object:
action: upload
overwrite: yes
bucket: gcs-bucket-name
src: binary.tar
dest: "binary.tar"
project: google-cloud-project-name
auth_kind: serviceaccount
service_account_file: "jwt.json"
state: present
```

Example playbook to upload text file to GCS:
```
# upload-text-file.yml
---

- name: Upload a text file to Google Cloud Storage bucket using gcp_storage_object module
hosts: all
connection: local
tasks:
- name: Upload Latest Backup(s) to GCS Bucket
gcp_storage_object:
action: upload
overwrite: yes
bucket: gcs-bucket-name
src: foobar.txt
dest: "foobar.txt"
project: google-cloud-project-name
auth_kind: serviceaccount
service_account_file: "jwt.json"
state: present
```

Run `upload-text-file.yml` using Python 2 (modify Python envs per your system):
```
ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=`which python2`
```

Run `upload-text-file.yml` using Python 3:
```
ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=`which python3`
```

Run `upload-bin-file.yml` using Python 2:
```
ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=`which python2`
```

Run `upload-bin-file.yml` using Python 3:
```
ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=`which python3`
```

##### EXPECTED RESULTS
It's expected that all files upload successfully to GCS, regardless of using Python 2, or Python 3 or file type.

##### ACTUAL RESULTS

Python 2 Text File Upload (successful):
```
$ ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2

PLAY [Upload a file to GCS using gcp_storage_object module] ********************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************
ok: [localhost]

TASK [Upload Latest Backup(s) to GCS Bucket] ***********************************************************************************************
changed: [localhost]

PLAY RECAP *********************************************************************************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
```

Python 3 Text File Upload (successful):
```
$ ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3

PLAY [Upload a file to GCS using gcp_storage_object module] *****************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [localhost]

TASK [Upload Latest Backup(s) to GCS Bucket] ********************************************************************************************************************************
changed: [localhost]

PLAY RECAP ******************************************************************************************************************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
```

Python 2 Binary File Upload (successful):
```
$ ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2

PLAY [Upload a file to GCS using gcp_storage_object module] *****************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [localhost]

TASK [Upload Latest Backup(s) to GCS Bucket] ********************************************************************************************************************************
changed: [localhost]

PLAY RECAP ******************************************************************************************************************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
```

Python 3 Binary File Upload (fails):
```
$ ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3 -vvvv
ansible-playbook 2.9.11
config file = None
configured module search path = ['${HOME}/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible
executable location = /Library/Frameworks/Python.framework/Versions/3.8/bin/ansible-playbook
python version = 3.8.5 (v3.8.5:580fbb018f, Jul 20 2020, 12:11:27) [Clang 6.0 (clang-600.0.57)]
No config file found; using defaults
setting up inventory plugins
Set default localhost to localhost
Parsed localhost, inventory source with host_list plugin
Loading callback plugin default of type stdout, v2.0 from /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible/plugins/callback/default.py

PLAYBOOK: upload-bin-file.yml ***********************************************************************************************************************************************************************************
Positional arguments: upload-bin-file.yml
verbosity: 4
connection: local
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('localhost,',)
extra_vars: ('ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3',)
forks: 5
1 plays in upload-bin-file.yml

PLAY [Upload a file to GCS using gcp_storage_object module] *****************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
task path: ${HOME}/Desktop/upload-bin-file.yml:3
ESTABLISH LOCAL CONNECTION FOR USER: bwallander
EXEC /bin/sh -c 'echo ~bwallander && sleep 0'
EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ${HOME}/.ansible/tmp `"&& mkdir ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500 && echo ansible-tmp-1596236353.566118-46540-87177398296500="` echo ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500 `" ) && sleep 0'
Using module file /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible/modules/system/setup.py
PUT ${HOME}/.ansible/tmp/ansible-local-46536i64et516/tmpj1i25bxy TO ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/AnsiballZ_setup.py
EXEC /bin/sh -c 'chmod u+x ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/ ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/AnsiballZ_setup.py && sleep 0'
EXEC /bin/sh -c '/Library/Frameworks/Python.framework/Versions/3.8/bin/python3 ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/AnsiballZ_setup.py && sleep 0'
EXEC /bin/sh -c 'rm -f -r ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/ > /dev/null 2>&1 && sleep 0'
ok: [localhost]
META: ran handlers

TASK [Upload Latest Backup(s) to GCS Bucket] ********************************************************************************************************************************************************************
task path: ${HOME}/Desktop/upload-bin-file.yml:7
ESTABLISH LOCAL CONNECTION FOR USER: bwallander
EXEC /bin/sh -c 'echo ~bwallander && sleep 0'
EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ${HOME}/.ansible/tmp `"&& mkdir ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804 && echo ansible-tmp-1596236354.939475-46572-195465935008804="` echo ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804 `" ) && sleep 0'
Using module file /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible/modules/cloud/google/gcp_storage_object.py
PUT ${HOME}/.ansible/tmp/ansible-local-46536i64et516/tmp1a31n78a TO ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py
EXEC /bin/sh -c 'chmod u+x ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/ ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py && sleep 0'
EXEC /bin/sh -c '/Library/Frameworks/Python.framework/Versions/3.8/bin/python3 ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py && sleep 0'
EXEC /bin/sh -c 'rm -f -r ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
File "${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py", line 102, in
_ansiballz_main()
File "${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py", line 94, in _ansiballz_main
invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
File "${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py", line 40, in invoke_module
runpy.run_module(mod_name='ansible.modules.cloud.google.gcp_storage_object', init_globals=None, run_name='__main__', alter_sys=True)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 207, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py", line 286, in
File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py", line 190, in main
File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py", line 206, in upload_file
File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/module_utils/gcp_utils.py", line 99, in post_contents
File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/module_utils/gcp_utils.py", line 159, in full_post
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/auth/transport/requests.py", line 448, in request
response = super(AuthorizedSession, self).request(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1039, in _send_output
for chunk in chunks:
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 994, in _read_readable
datablock = readable.read(self.blocksize)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x98 in position 522: invalid start byte
fatal: [localhost]: FAILED! => {
"changed": false,
"module_stderr": "Traceback (most recent call last):\n File \"${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py\", line 102, in \n _ansiballz_main()\n File \"${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py\", line 94, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py\", line 40, in invoke_module\n runpy.run_module(mod_name='ansible.modules.cloud.google.gcp_storage_object', init_globals=None, run_name='__main__', alter_sys=True)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py\", line 207, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py\", line 286, in \n File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py\", line 190, in main\n File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py\", line 206, in upload_file\n File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/module_utils/gcp_utils.py\", line 99, in post_contents\n File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/module_utils/gcp_utils.py\", line 159, in full_post\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py\", line 578, in post\n return self.request('POST', url, data=data, json=json, **kwargs)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/auth/transport/requests.py\", line 448, in request\n response = super(AuthorizedSession, self).request(\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py\", line 530, in request\n resp = self.send(prep, **send_kwargs)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py\", line 643, in send\n r = adapter.send(request, **kwargs)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/adapters.py\", line 439, in send\n resp = conn.urlopen(\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 670, in urlopen\n httplib_response = self._make_request(\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 392, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 1255, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 1301, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 1250, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 1039, in _send_output\n for chunk in chunks:\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 994, in _read_readable\n datablock = readable.read(self.blocksize)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/codecs.py\", line 322, in decode\n (result, consumed) = self._buffer_decode(data, self.errors, final)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0x98 in position 522: invalid start byte\n",
"module_stdout": "",
"msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
"rc": 1
}

PLAY RECAP ******************************************************************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
```

_Copied from original issue: ansible/ansible#71034_

Start at the gcp_storage_object module entry point and reproduce the failure with the supplied upload-bin-file.yml playbook, binary.tar, and Python 3. Compare it with the successful text-file and Python 2 runs. Done means binary files upload successfully under Python 3 without the reported UnicodeDecodeError.

Written by the indexing model from the issue text.