[BUG] Error in function _file_list
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
2022-06-22 14:03:01,018 [salt.master :1984][ERROR ][947] Error in function _file_list:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/payload.py", line 196, in dumps
msg, default=ext_type_encoder, use_bin_type=use_bin_type
File "/usr/lib/python3.6/site-packages/salt/utils/msgpack.py", line 133, in packb
return msgpack.packb(o, **_sanitize_msgpack_kwargs(kwargs))
File "/usr/lib64/python3.6/site-packages/msgpack/init.py", line 46, in packb
return Packer(**kwargs).pack(o)
File "msgpack/_packer.pyx", line 284, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 290, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 287, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 91-102: surrogates not allowed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/master.py", line 1978, in run_func
ret = getattr(self, func)(load)
File "/usr/lib/python3.6/site-packages/salt/utils/decorators/init.py", line 791, in wrapped
return function(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/fileserver/init.py", line 775, in file_list
ret.update(self.serversfstr)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1241, in call
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2274, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/init.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2289, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/fileserver/roots.py", line 432, in file_list
return _file_lists(load, "files")
File "/usr/lib/python3.6/site-packages/salt/fileserver/roots.py", line 418, in file_lists
salt.fileserver.write_file_list_cache(opts, ret, list_cache, w_lock)
File "/usr/lib/python3.6/site-packages/salt/fileserver/init.py", line 180, in write_file_list_cache
fp.write(serial.dumps(data))
File "/usr/lib/python3.6/site-packages/salt/payload.py", line 233, in dumps
msg, default=ext_type_encoder, use_bin_type=use_bin_type
File "/usr/lib/python3.6/site-packages/salt/utils/msgpack.py", line 133, in packb
return msgpack.packb(o, **_sanitize_msgpack_kwargs(kwargs))
File "/usr/lib64/python3.6/site-packages/msgpack/init.py", line 46, in packb
return Packer(**kwargs).pack(o)
File "msgpack/_packer.pyx", line 284, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 290, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 287, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 91-102: surrogates not allowed
2022-06-22 14:03:10,283 [salt.master :492 ][ERROR ][19071] Uncaught exception while updating roots fileserver cache
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/master.py", line 488, in _do_update
update_func(*args)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1241, in call
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2274, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/init.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2289, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/fileserver/roots.py", line 194, in update
salt.utils.stringutils.to_bytes("{}:{}\n".format(file_path, mtime))
File "/usr/lib/python3.6/site-packages/salt/utils/stringutils.py", line 53, in to_bytes
raise exc # pylint: disable=raising-bad-type
File "/usr/lib/python3.6/site-packages/salt/utils/stringutils.py", line 46, in to_bytes
return s.encode(enc, errors)
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 106-117: surrogates not allowed
Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Please be as specific as possible and give set-up details.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
Steps to Reproduce the behavior
(Include debug logs if possible and relevant)
update_lua_nb:
file.recurse:
- name: /usr/local/ngx_openresty_proxy/nginx/lua/nb
- source: salt://install_soft/vod/scdn_sched/files/ngx_openresty_proxy/nginx/lua/nb
- user: nobody
- group: nobody
- file_mode: 644
- dir_mode: 755
- recurse:
- user
- group
- makedirs: true
- include_enpty: true
- keep_symlinks: true
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) /usr/bin/salt-master -V Salt Version: Salt: 3003.5Dependency Versions:
cffi: 1.14.1
cherrypy: 18.6.0
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: 3.14.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Apr 2 2020, 13:34:55)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 17.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1127.13.1.el7.x86_64
system: Linux
version: CentOS Linux 7 Core
PASTE HERE
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the file.recurse state and trace the failure through salt/fileserver/roots.py (_file_lists and update), salt/fileserver/init.py (write_file_list_cache), salt/payload.py (dumps), and salt/utils/stringutils.py (to_bytes). Done means the fileserver cache update and file listing handle the reported surrogate characters without either UnicodeEncodeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100