s3fs appears to double encode urls when fetching
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
S3 urls when fetching for s3fs appear to be double encoded. I have an item in a bucket containing a + symbol that is being rendered as %252B. Which is + -> %2B -> %252B.
Setup
fileserver_backend:
- s3fs
etc..
Steps to Reproduce Issue
Have an item in a bucket with a +. Running salt-run fileserver.update should yield a log line somewhere like
[INFO ] example - base : examplel/unstable/example_0.1.1-alpha.1+57dfd57_armhf.deb
[DEBUG ] S3 Request: http://example.s3.amazonaws.com/example/unstable/example_0.1.1-alpha.1%252B57dfd57_armhf.deb?
The caching will blow up with a 404, which is not surprising. Note how the + has been encoded. Also note that salt got the listing correct, just not the fetching.
Versions Report
Salt Version:
Salt: 2019.2.0
Dependency Versions:
cffi: 1.6.0
cherrypy: Not Installed
dateutil: 1.5
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: 0.26.3
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.26.4
Python: 2.7.5 (default, Apr 9 2019, 14:30:50)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: centos 7.6.1810 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-957.1.3.el7.x86_64
system: Linux
version: CentOS Linux 7.6.1810 Core
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the s3fs fileserver backend and reproduce the issue with salt-run fileserver.update using an S3 object whose name contains +. Inspect the S3 request shown in the logs; done means the fetch uses the correctly encoded object path and no longer fails with a 404.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100