[BUG] salt-syndic fail to properly switch to different user with log file:///dev/log
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
Trying to run salt-syndic as a salt user (along with salt-master also running with that user).
Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Salt (master and syndic) configured to run as user salt.
# /etc/salt/master.d/f_defaults.conf
user: "salt"
log_file: "file:///dev/log"
syndic_log_file: "file:///dev/log"
syndic_user: "salt"
Please be as specific as possible and give set-up details.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service - AWS EC2, but most likely not relevant
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
Steps to Reproduce the behavior
Configure salt-master and syndic to run as user.
Launching as root directly
# salt-syndic
[ERROR ] Unable to connect pusher: Stream is closed
[ERROR ] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x78f86b0c48b0>, <salt.ext.tornado.concurrent.Future object at 0x78f86a1f62e0>)
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 423, in connect_pull
self.pusher.connect(timeout=timeout)
File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/transport/ipc.py", line 342, in _connect
yield self.stream.connect(sock_addr)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
ret = callback()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
return fn(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1070, in run
yielded = self.gen.send(value)
File "/usr/lib/python3/dist-packages/salt/crypt.py", line 710, in _authenticate
event.fire_event(
File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 799, in fire_event
if not self.connect_pull(timeout=timeout_s):
File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 426, in connect_pull
log.error(
File "/usr/lib/python3.9/logging/__init__.py", line 1471, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/_logging/impl.py", line 301, in _log
LOGGING_LOGGER_CLASS._log(
File "/usr/lib/python3.9/logging/__init__.py", line 1585, in _log
self.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1595, in handle
self.callHandlers(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1657, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 948, in handle
self.emit(record)
File "/usr/lib/python3.9/logging/handlers.py", line 491, in emit
self.reopenIfNeeded()
File "/usr/lib/python3.9/logging/handlers.py", line 470, in reopenIfNeeded
sres = os.stat(self.baseFilename)
PermissionError: [Errno 13] Permission denied: '/root/.salt/syndic.log'
[ERROR ] An un-handled exception was caught by salt's global exception handler:
PermissionError: [Errno 13] Permission denied: '/root/.salt/syndic.log'
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 423, in connect_pull
self.pusher.connect(timeout=timeout)
File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/transport/ipc.py", line 342, in _connect
yield self.stream.connect(sock_addr)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
ret = callback()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
return fn(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1070, in run
yielded = self.gen.send(value)
File "/usr/lib/python3/dist-packages/salt/crypt.py", line 710, in _authenticate
event.fire_event(
File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 799, in fire_event
if not self.connect_pull(timeout=timeout_s):
File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 426, in connect_pull
log.error(
File "/usr/lib/python3.9/logging/__init__.py", line 1471, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/_logging/impl.py", line 301, in _log
LOGGING_LOGGER_CLASS._log(
File "/usr/lib/python3.9/logging/__init__.py", line 1585, in _log
self.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1595, in handle
self.callHandlers(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1657, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 948, in handle
self.emit(record)
File "/usr/lib/python3.9/logging/handlers.py", line 491, in emit
self.reopenIfNeeded()
File "/usr/lib/python3.9/logging/handlers.py", line 470, in reopenIfNeeded
sres = os.stat(self.baseFilename)
PermissionError: [Errno 13] Permission denied: '/root/.salt/syndic.log'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/salt-syndic", line 33, in <module>
sys.exit(load_entry_point('salt==3004', 'console_scripts', 'salt-syndic')())
File "/usr/lib/python3/dist-packages/salt/scripts.py", line 390, in salt_syndic
syndic.start()
File "/usr/lib/python3/dist-packages/salt/cli/daemons.py", line 619, in start
self.syndic.tune_in()
File "/usr/lib/python3/dist-packages/salt/minion.py", line 3600, in tune_in
self.io_loop.start()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 834, in start
self._run_callback(self._callbacks.popleft())
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 624, in _run_callback
self.handle_callback_exception(callback)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 640, in handle_callback_exception
app_log.error("Exception in callback %r", callback, exc_info=True)
File "/usr/lib/python3.9/logging/__init__.py", line 1471, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/_logging/impl.py", line 301, in _log
LOGGING_LOGGER_CLASS._log(
File "/usr/lib/python3.9/logging/__init__.py", line 1585, in _log
self.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1595, in handle
self.callHandlers(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1657, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 948, in handle
self.emit(record)
File "/usr/lib/python3.9/logging/handlers.py", line 491, in emit
self.reopenIfNeeded()
File "/usr/lib/python3.9/logging/handlers.py", line 470, in reopenIfNeeded
sres = os.stat(self.baseFilename)
PermissionError: [Errno 13] Permission denied: '/root/.salt/syndic.log'
An un-handled exception was caught by salt's global exception handler:
PermissionError: [Errno 13] Permission denied: '/root/.salt/syndic.log'
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 423, in connect_pull
self.pusher.connect(timeout=timeout)
File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/transport/ipc.py", line 342, in _connect
yield self.stream.connect(sock_addr)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
ret = callback()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
return fn(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1070, in run
yielded = self.gen.send(value)
File "/usr/lib/python3/dist-packages/salt/crypt.py", line 710, in _authenticate
event.fire_event(
File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 799, in fire_event
if not self.connect_pull(timeout=timeout_s):
File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 426, in connect_pull
log.error(
File "/usr/lib/python3.9/logging/__init__.py", line 1471, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/_logging/impl.py", line 301, in _log
LOGGING_LOGGER_CLASS._log(
File "/usr/lib/python3.9/logging/__init__.py", line 1585, in _log
self.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1595, in handle
self.callHandlers(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1657, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 948, in handle
self.emit(record)
File "/usr/lib/python3.9/logging/handlers.py", line 491, in emit
self.reopenIfNeeded()
File "/usr/lib/python3.9/logging/handlers.py", line 470, in reopenIfNeeded
sres = os.stat(self.baseFilename)
PermissionError: [Errno 13] Permission denied: '/root/.salt/syndic.log'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/salt-syndic", line 33, in <module>
sys.exit(load_entry_point('salt==3004', 'console_scripts', 'salt-syndic')())
File "/usr/lib/python3/dist-packages/salt/scripts.py", line 390, in salt_syndic
syndic.start()
File "/usr/lib/python3/dist-packages/salt/cli/daemons.py", line 619, in start
self.syndic.tune_in()
File "/usr/lib/python3/dist-packages/salt/minion.py", line 3600, in tune_in
self.io_loop.start()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 834, in start
self._run_callback(self._callbacks.popleft())
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 624, in _run_callback
self.handle_callback_exception(callback)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 640, in handle_callback_exception
app_log.error("Exception in callback %r", callback, exc_info=True)
File "/usr/lib/python3.9/logging/__init__.py", line 1471, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/_logging/impl.py", line 301, in _log
LOGGING_LOGGER_CLASS._log(
File "/usr/lib/python3.9/logging/__init__.py", line 1585, in _log
self.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1595, in handle
self.callHandlers(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1657, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 948, in handle
self.emit(record)
File "/usr/lib/python3.9/logging/handlers.py", line 491, in emit
self.reopenIfNeeded()
File "/usr/lib/python3.9/logging/handlers.py", line 470, in reopenIfNeeded
sres = os.stat(self.baseFilename)
PermissionError: [Errno 13] Permission denied: '/root/.salt/syndic.log'
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 423, in connect_pull
self.pusher.connect(timeout=timeout)
File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/transport/ipc.py", line 342, in _connect
yield self.stream.connect(sock_addr)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
ret = callback()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
return fn(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1070, in run
yielded = self.gen.send(value)
File "/usr/lib/python3/dist-packages/salt/crypt.py", line 710, in _authenticate
event.fire_event(
File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 799, in fire_event
if not self.connect_pull(timeout=timeout_s):
File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 426, in connect_pull
log.error(
File "/usr/lib/python3.9/logging/__init__.py", line 1471, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/_logging/impl.py", line 301, in _log
LOGGING_LOGGER_CLASS._log(
File "/usr/lib/python3.9/logging/__init__.py", line 1585, in _log
self.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1595, in handle
self.callHandlers(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1657, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 948, in handle
self.emit(record)
File "/usr/lib/python3.9/logging/handlers.py", line 491, in emit
self.reopenIfNeeded()
File "/usr/lib/python3.9/logging/handlers.py", line 470, in reopenIfNeeded
sres = os.stat(self.baseFilename)
PermissionError: [Errno 13] Permission denied: '/root/.salt/syndic.log'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/salt-syndic", line 33, in <module>
sys.exit(load_entry_point('salt==3004', 'console_scripts', 'salt-syndic')())
File "/usr/lib/python3/dist-packages/salt/scripts.py", line 390, in salt_syndic
syndic.start()
File "/usr/lib/python3/dist-packages/salt/cli/daemons.py", line 619, in start
self.syndic.tune_in()
File "/usr/lib/python3/dist-packages/salt/minion.py", line 3600, in tune_in
self.io_loop.start()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 834, in start
self._run_callback(self._callbacks.popleft())
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 624, in _run_callback
self.handle_callback_exception(callback)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 640, in handle_callback_exception
app_log.error("Exception in callback %r", callback, exc_info=True)
File "/usr/lib/python3.9/logging/__init__.py", line 1471, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/_logging/impl.py", line 301, in _log
LOGGING_LOGGER_CLASS._log(
File "/usr/lib/python3.9/logging/__init__.py", line 1585, in _log
self.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1595, in handle
self.callHandlers(record)
File "/usr/lib/python3.9/logging/__init__.py", line 1657, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.9/logging/__init__.py", line 948, in handle
self.emit(record)
File "/usr/lib/python3.9/logging/handlers.py", line 491, in emit
self.reopenIfNeeded()
File "/usr/lib/python3.9/logging/handlers.py", line 470, in reopenIfNeeded
sres = os.stat(self.baseFilename)
PermissionError: [Errno 13] Permission denied: '/root/.salt/syndic.log'
While directly launching it as user would work:
# sudo -u salt salt-syndic
[ERROR ] Unable to connect pusher: Stream is closed
[ERROR ] Unable to call _fire_master on 10.x.y.z, that syndic is not connected
[CRITICAL] Unable to call _fire_master on any masters!
[ERROR ] Unable to connect pusher: Stream is closed
[ERROR ] Unable to connect pusher: Stream is closed
[ERROR ] Unable to connect pusher: Stream is closed
[ERROR ] Unable to connect pusher: Stream is closed
^C[WARNING ] Syndic received a SIGINT. Exiting.
The Salt Syndic is shutdown. Syndic received a SIGINT. Exited.
Expected behavior
I would expect the salt-syndic to properly run as the salt user, and not try to access files in /root.
Might be nice to load files that should only be accessible by root (like minion key, as minion is running as root), but in any case not force to access files that have no reason to be there anyway later.
Screenshots
N/A
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Debian package provided on salt repositories:
salt-syndic3004+ds-1
Salt Version:
Salt: 3004
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.3
libgit2: Not Installed
M2Crypto: 0.37.1
Mako: Not Installed
msgpack: 1.0.0
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: 3.9.7
pygit2: Not Installed
Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
python-gnupg: Not Installed
PyYAML: 5.3.1
PyZMQ: 20.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: debian 11 bullseye
locale: utf-8
machine: x86_64
release: 5.10.41-grsec-cerberhost-0.17.0-rap
system: Linux
version: Debian GNU/Linux 11 bullseye
Additional context
There might also be a problem about program-name while running directly as user salt, but I haven't looked much more into it so far.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the syndic startup path in salt/cli/daemons.py and the event connection handling in salt/utils/event.py, then reproduce the issue using the listed user and logging configuration. Trace why startup reaches /root/.salt/syndic.log despite the configured user and file:///dev/log. Done means salt-syndic runs as salt without the reported permission failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100