[BUG] salt-run fails to cache files from S3 with space or () symbols in name/path
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
salt-run fails to cache files from AWS S3 with space or () symbols in name/path, e.g.: icon_Alert(14x13).png or Autodesk Shared
Setup
/etc/salt/master:
s3.keyid: mykeyid
s3.key: mykey
s3.location: eu-central-1
s3.https_enable: True
s3.buckets:
base:
- mybucket
Steps to Reproduce the behaviour
Run (with icon_Alert(14x13).png or Autodesk Shared files at s3://mybucket/base/ location) :
sudo salt-run fileserver.update backend=s3fs -l debug
Output:
[INFO ] mybucket - base : base/autodesk/3dsmax/2018/Img/x64/CLIC/Autodesk/CLM/V5/resources/ipm_button/icon_Alert(14x13).png
[DEBUG ] LazyLoaded s3.query
[DEBUG ] S3 Request: https://mybucket.s3.amazonaws.com/base/autodesk/3dsmax/2018/Img/x64/CLIC/Autodesk/CLM/V5/resources/ipm_button/icon_Alert%252814x13%2529.png?
[DEBUG ] S3 Headers::
[DEBUG ] Authorization: AWS4-HMAC-SHA256 Credential=mykeyid/20210116/eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=5bfef5978b317f2cebc205a121f99a94b30d59aeb4c6c10bcdc1c14dd9f92437
[DEBUG ] Starting new HTTPS connection (1): mybucket.s3.amazonaws.com:443
[DEBUG ] https://mybucket.s3.amazonaws.com:443 "GET /base/autodesk/3dsmax/2018/Img/x64/CLIC/Autodesk/CLM/V5/resources/ipm_button/icon_Alert%252814x13%2529.png HTTP/1.1" 404 None
[DEBUG ] Response content: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>base/autodesk/3dsmax/2018/Img/x64/CLIC/Autodesk/CLM/V5/resources/ipm_button/icon_Alert%2814x13%29.png</Key><RequestId>32ADE1644D09C0D8</RequestId><HostId>4/99nZdNWMtVwTw3o98nv+44CUCHxKis3uu64OHPmuTzrTtb1CV1UGERtjNecRwvmlNBA2iGqbE=</HostId></Error>
[DEBUG ] S3 Response Status Code: 404
[DEBUG ] LazyLoaded local_cache.prep_jid
[DEBUG ] Sending event: tag = salt/run/20210116185552652456/ret; data = {'fun': 'runner.fileserver.update', 'jid': '20210116185552652456', 'user': 'sudo_ubuntu', 'fun_args': [{'backend': 's3fs'}], '_stamp': '2021-01-16T18:55:54.000742', 'return': 'Exception occurred in runner fileserver.update: Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/salt/client/mixins.py", line 397, in low\n data["return"] = func(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/salt/runners/fileserver.py", line 352, in update\n fileserver.update(back=backend)\n File "/usr/lib/python3/dist-packages/salt/fileserver/__init__.py", line 492, in update\n self.servers[fstr]()\n File "/usr/lib/python3/dist-packages/salt/fileserver/s3fs.py", line 143, in update\n _get_file_from_s3(\n File "/usr/lib/python3/dist-packages/salt/fileserver/s3fs.py", line 783, in _get_file_from_s3\n __utils__["s3.query"](\n File "/usr/lib/python3/dist-packages/salt/utils/s3.py", line 279, in query\n raise CommandExecutionError(\nsalt.exceptions.CommandExecutionError: Failed to get file. NoSuchKey: The specified key does not exist.\n', 'success': False}
[DEBUG ] LazyLoaded nested.output
Exception occurred in runner fileserver.update: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/client/mixins.py", line 397, in low
data["return"] = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/runners/fileserver.py", line 352, in update
fileserver.update(back=backend)
File "/usr/lib/python3/dist-packages/salt/fileserver/__init__.py", line 492, in update
self.servers[fstr]()
File "/usr/lib/python3/dist-packages/salt/fileserver/s3fs.py", line 143, in update
_get_file_from_s3(
File "/usr/lib/python3/dist-packages/salt/fileserver/s3fs.py", line 783, in _get_file_from_s3
__utils__["s3.query"](
File "/usr/lib/python3/dist-packages/salt/utils/s3.py", line 279, in query
raise CommandExecutionError(
salt.exceptions.CommandExecutionError: Failed to get file. NoSuchKey: The specified key does not exist.
[INFO ] Runner completed: 20210116185552652456
[DEBUG ] Closing IPCMessageClient instance
[DEBUG ] Runner return: Exception occurred in runner fileserver.update: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/client/mixins.py", line 397, in low
data["return"] = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/runners/fileserver.py", line 352, in update
fileserver.update(back=backend)
File "/usr/lib/python3/dist-packages/salt/fileserver/__init__.py", line 492, in update
self.servers[fstr]()
File "/usr/lib/python3/dist-packages/salt/fileserver/s3fs.py", line 143, in update
_get_file_from_s3(
File "/usr/lib/python3/dist-packages/salt/fileserver/s3fs.py", line 783, in _get_file_from_s3
__utils__["s3.query"](
File "/usr/lib/python3/dist-packages/salt/utils/s3.py", line 279, in query
raise CommandExecutionError(
salt.exceptions.CommandExecutionError: Failed to get file. NoSuchKey: The specified key does not exist.
Expected behaviour
Files with special symbols and spaces should be cached from s3.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3001.4
Dependency Versions:
cffi: 1.14.4
cherrypy: unknown
dateutil: 2.7.3
docker-py: Not Installed
gitdb: 2.0.6
gitpython: 3.0.7
Jinja2: 2.10.1
libgit2: 1.1.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: 2.20
pycrypto: 2.6.1
pycryptodome: 3.6.1
pygit2: 1.4.0
Python: 3.8.5 (default, Jul 28 2020, 12:59:40)
python-gnupg: 0.4.5
PyYAML: 5.3.1
PyZMQ: 18.1.1
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2
System Versions:
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-1029-aws
system: Linux
version: Ubuntu 20.04 focal
Additional context
I have the same issue with saltstack ver 3002.2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with salt/fileserver/s3fs.py, especially _get_file_from_s3, and follow its call into salt/utils/s3.py query. Reproduce with salt-run fileserver.update backend=s3fs -l debug using an S3 key containing spaces or parentheses; done means those files are cached instead of producing a NoSuchKey error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100