saltstack / saltstack/salt

s3fs appears to double encode urls when fetching

Open
#53,539 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

S3 urls when fetching for s3fs appear to be double encoded. I have an item in a bucket containing a + symbol that is being rendered as %252B. Which is + -> %2B -> %252B.

Setup

fileserver_backend:

  • s3fs

etc..

Steps to Reproduce Issue

Have an item in a bucket with a +. Running salt-run fileserver.update should yield a log line somewhere like

[INFO    ] example - base : examplel/unstable/example_0.1.1-alpha.1+57dfd57_armhf.deb
[DEBUG   ] S3 Request: http://example.s3.amazonaws.com/example/unstable/example_0.1.1-alpha.1%252B57dfd57_armhf.deb?

The caching will blow up with a 404, which is not surprising. Note how the + has been encoded. Also note that salt got the listing correct, just not the fetching.

Versions Report

Salt Version:
Salt: 2019.2.0

Dependency Versions:
cffi: 1.6.0
cherrypy: Not Installed
dateutil: 1.5
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: 0.26.3
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.26.4
Python: 2.7.5 (default, Apr 9 2019, 14:30:50)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.6.1810 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-957.1.3.el7.x86_64
system: Linux
version: CentOS Linux 7.6.1810 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the s3fs fileserver backend and reproduce the issue with salt-run fileserver.update using an S3 object whose name contains +. Inspect the S3 request shown in the logs; done means the fetch uses the correctly encoded object path and no longer fails with a 404.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.