saltstack / saltstack/salt

[BUG] Error in function _file_list

Open
#62,201 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Core needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
2022-06-22 14:03:01,018 [salt.master :1984][ERROR ][947] Error in function _file_list:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/payload.py", line 196, in dumps
msg, default=ext_type_encoder, use_bin_type=use_bin_type
File "/usr/lib/python3.6/site-packages/salt/utils/msgpack.py", line 133, in packb
return msgpack.packb(o, **_sanitize_msgpack_kwargs(kwargs))
File "/usr/lib64/python3.6/site-packages/msgpack/init.py", line 46, in packb
return Packer(**kwargs).pack(o)
File "msgpack/_packer.pyx", line 284, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 290, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 287, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 91-102: surrogates not allowed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/master.py", line 1978, in run_func
ret = getattr(self, func)(load)
File "/usr/lib/python3.6/site-packages/salt/utils/decorators/init.py", line 791, in wrapped
return function(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/fileserver/init.py", line 775, in file_list
ret.update(self.serversfstr)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1241, in call
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2274, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/init.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2289, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/fileserver/roots.py", line 432, in file_list
return _file_lists(load, "files")
File "/usr/lib/python3.6/site-packages/salt/fileserver/roots.py", line 418, in file_lists
salt.fileserver.write_file_list_cache(opts, ret, list_cache, w_lock)
File "/usr/lib/python3.6/site-packages/salt/fileserver/init.py", line 180, in write_file_list_cache
fp
.write(serial.dumps(data))
File "/usr/lib/python3.6/site-packages/salt/payload.py", line 233, in dumps
msg, default=ext_type_encoder, use_bin_type=use_bin_type
File "/usr/lib/python3.6/site-packages/salt/utils/msgpack.py", line 133, in packb
return msgpack.packb(o, **_sanitize_msgpack_kwargs(kwargs))
File "/usr/lib64/python3.6/site-packages/msgpack/init.py", line 46, in packb
return Packer(**kwargs).pack(o)
File "msgpack/_packer.pyx", line 284, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 290, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 287, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 91-102: surrogates not allowed
2022-06-22 14:03:10,283 [salt.master :492 ][ERROR ][19071] Uncaught exception while updating roots fileserver cache
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/master.py", line 488, in _do_update
update_func(*args)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1241, in call
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2274, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/init.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2289, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/fileserver/roots.py", line 194, in update
salt.utils.stringutils.to_bytes("{}:{}\n".format(file_path, mtime))
File "/usr/lib/python3.6/site-packages/salt/utils/stringutils.py", line 53, in to_bytes
raise exc # pylint: disable=raising-bad-type
File "/usr/lib/python3.6/site-packages/salt/utils/stringutils.py", line 46, in to_bytes
return s.encode(enc, errors)
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 106-117: surrogates not allowed

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)
update_lua_nb:
file.recurse:
- name: /usr/local/ngx_openresty_proxy/nginx/lua/nb
- source: salt://install_soft/vod/scdn_sched/files/ngx_openresty_proxy/nginx/lua/nb
- user: nobody
- group: nobody
- file_mode: 644
- dir_mode: 755
- recurse:
- user
- group
- makedirs: true
- include_enpty: true
- keep_symlinks: true

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) /usr/bin/salt-master -V Salt Version: Salt: 3003.5

Dependency Versions:
cffi: 1.14.1
cherrypy: 18.6.0
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: 3.14.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Apr 2 2020, 13:34:55)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 17.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4

System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1127.13.1.el7.x86_64
system: Linux
version: CentOS Linux 7 Core

PASTE HERE

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the file.recurse state and trace the failure through salt/fileserver/roots.py (_file_lists and update), salt/fileserver/init.py (write_file_list_cache), salt/payload.py (dumps), and salt/utils/stringutils.py (to_bytes). Done means the fileserver cache update and file listing handle the reported surrogate characters without either UnicodeEncodeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.