salt-cp - UnicodeEncodeError in files with non-ascii characters
Open
Nobody has claimed this yet.
bug
severity-low
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
Using salt-cp to copy a file with non-ascii characters causes a decode error (and an incomplete copy).
Setup
Salt: 2019.2.0
Steps to Reproduce Issue
Setting up for the test...
# mkdir -p /root/test
# cd /root/test
# rm -f /root/test/*
# curl -s https://www.w3.org/2001/06/utf-8-test/UTF-8-demo.html -o utf8.txt
# salt ip-10-0-253-180-centos1 test.ping
ip-10-0-253-180-centos1:
True
Ensure environment is sane...
# systemctl restart salt-api salt-master
# salt ip-10-0-253-180-centos1 test.ping
ip-10-0-253-180-centos1:
True
Reproduce issue....
# salt-cp ip-10-0-253-180-centos1 utf8.txt /tmp/
ip-10-0-253-180-centos1:
The minion function caused an exception: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1607, in _thread_return
return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
File "/usr/lib/python2.7/site-packages/salt/executors/direct_call.py", line 12, in execute
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/salt/modules/cp.py", line 84, in recv
fp_.write(data)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 347-382: ordinal not in range(128)
And from /var/log/salt/master...
2019-06-14 15:29:24,503 [salt.master :1577][ERROR ][15675] Received minion error from [ip-10-0-253-180-centos1]: The minion function caused an exception
Versions Report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
# salt-master --versions
/usr/lib/python2.7/site-packages/salt/scripts.py:102: DeprecationWarning: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. Salt will drop support for Python 2.7 in the Sodium release or later.
Salt Version:
Salt: 2019.2.0
Dependency Versions:
cffi: 1.6.0
cherrypy: 5.6.0
dateutil: 2.8.0
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: 0.26.3
libnacl: Not Installed
M2Crypto: 0.31.0
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.26.4
Python: 2.7.5 (default, Apr 9 2019, 14:30:50)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 15.3.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: centos 7.6.1810 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-957.1.3.el7.x86_64
system: Linux
version: CentOS Linux 7.6.1810 Core
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with salt/modules/cp.py and its recv function, where the traceback shows the failure at fp_.write(data). Reproduce the salt-cp command using the provided UTF-8 demo file, then verify that the non-ASCII file is copied completely without a UnicodeEncodeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100