saltstack / saltstack/salt

encoding error with gitfs and files with different encoding from global one

Open
#56,521 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Confirmed File Servers severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

with salt 3000 on python 3 and a gitfs source, if you try to use a path from git containing a text file in an encoding different from the system one, but that contains valid utf8 (like ascii) in its first 2k, it fails with an error similar to this one:

2020-04-03 19:53:22,663 [salt.master      :1846][ERROR   ][61192] Error in function _serve_file:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/master.py", line 1839, in run_func
    ret = getattr(self, func)(load)
  File "/usr/lib/python3.6/site-packages/salt/fileserver/__init__.py", line 650, in serve_file
    return self.servers[fstr](load, fnd)
  File "/usr/lib/python3.6/site-packages/salt/fileserver/gitfs.py", line 170, in serve_file
    return _gitfs().serve_file(load, fnd)
  File "/usr/lib/python3.6/site-packages/salt/utils/gitfs.py", line 2861, in serve_file
    data = data.decode(__salt_system_encoding__)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 4193: invalid continuation byte
Versions Report
Salt Version:
           Salt: 3000.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 2.8.1
      docker-py: Not Installed
          gitdb: 2.0.6
      gitpython: 3.0.5
         Jinja2: 2.10.3
        libgit2: Not Installed
       M2Crypto: 0.33.0
           Mako: 1.1.0
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: 1.4.5
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Aug  7 2019, 17:28:10)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 15.3.0
          smmap: 2.0.5
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.7.1908 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1062.4.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.7.1908 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with a gitfs text file whose encoding differs from the system encoding but begins with valid UTF-8. Start at salt/utils/gitfs.py in serve_file(), around the reported decode at line 2861, and trace how the file data is decoded. Done means gitfs serves the file without the UnicodeDecodeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.