encoding error with gitfs and files with different encoding from global one
Open
Nobody has claimed this yet.
bug
Confirmed
File Servers
severity-medium
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
with salt 3000 on python 3 and a gitfs source, if you try to use a path from git containing a text file in an encoding different from the system one, but that contains valid utf8 (like ascii) in its first 2k, it fails with an error similar to this one:
2020-04-03 19:53:22,663 [salt.master :1846][ERROR ][61192] Error in function _serve_file:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/master.py", line 1839, in run_func
ret = getattr(self, func)(load)
File "/usr/lib/python3.6/site-packages/salt/fileserver/__init__.py", line 650, in serve_file
return self.servers[fstr](load, fnd)
File "/usr/lib/python3.6/site-packages/salt/fileserver/gitfs.py", line 170, in serve_file
return _gitfs().serve_file(load, fnd)
File "/usr/lib/python3.6/site-packages/salt/utils/gitfs.py", line 2861, in serve_file
data = data.decode(__salt_system_encoding__)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 4193: invalid continuation byte
Versions Report
Salt Version:
Salt: 3000.1
Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: 2.0.6
gitpython: 3.0.5
Jinja2: 2.10.3
libgit2: Not Installed
M2Crypto: 0.33.0
Mako: 1.1.0
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: 1.4.5
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Aug 7 2019, 17:28:10)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 15.3.0
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7.7.1908 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1062.4.3.el7.x86_64
system: Linux
version: CentOS Linux 7.7.1908 Core
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with a gitfs text file whose encoding differs from the system encoding but begins with valid UTF-8. Start at salt/utils/gitfs.py in serve_file(), around the reported decode at line 2861, and trace how the file data is decoded. Done means gitfs serves the file without the UnicodeDecodeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100