saltstack / saltstack/salt

[BUG] usage of external_auth and rest_cherrypy is mutually exclusive in 3007.1

Open
#66,638 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
In Salt 3007.1 external_auth requires ipc_mode: tcp which is breaking functionality of salt-api

I need to setup external_auth on Salt Master 3007.1, which is not working properly without setting ipc_mode: tcp as I've found in this blog.

If it's intended change, it would be great if official documentation was also updated!

Setup
Salt Master configuration:
/etc/salt/master.d/run_user.conf

user: root

/etc/salt/master.d/external_auth.conf

external_auth:
  pam:
    user:
      - '*':
        - test.ping
        - network.*
        - event.fire
    salt-api:
      - .*
      - '@wheel'
      - '@runner'

ipc_mode: tcp

/etc/salt/master.d/cherrypy_ssl.conf

rest_cherrypy:
  debug: True
  port: 8000
  ssl_crt: /etc/pki/tls/certs/saltapi.crt
  ssl_key: /etc/pki/tls/certs/saltapi.key

netapi_enable_clients:
  - local
  - local_async
  - local_batch
  - local_subset
  - runner
  - runner_async

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM - Virtualbox
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
Setting ipc_mode: tcp is causing that after clean start of service (after reboot) are socket files not created:

root@sm-3007:~# ls -la /var/run/salt/master/
total 0
drwxr-xr-x 2 root root 40 Jun 14 10:21 .
drwxr-xr-x 4 root root 80 Jun 14 08:01 ..

If these files are missing, salt-api is not able to talk to Salt Master service, and therefor all salt-api calls are getting 503 ..

user@sm-3007:~$ salt-pepper '*' test.version
Error with request: HTTP Error 503: Service Unavailable
ERROR 2024-06-14 09:59:04,940 libpepper: Error with request: HTTP Error 503: Service Unavailable
HTTP Error 503: Service Unavailable
Uncaught Pepper error (increase verbosity for the full traceback).

server side:

[INFO    ] [14/Jun/2024:09:58:51] ENGINE Serving on https://0.0.0.0:8000                               
[14/Jun/2024:09:58:51] ENGINE Bus STARTED                                                              
[INFO    ] [14/Jun/2024:09:58:51] ENGINE Bus STARTED                                                   
127.0.0.1 - - [14/Jun/2024:09:59:04] "POST /login HTTP/1.1" 503 2184 "" "Python-urllib/3.10"           
[INFO    ] 127.0.0.1 - - [14/Jun/2024:09:59:04] "POST /login HTTP/1.1" 503 2184 "" "Python-urllib/3.10"

In case that ipc_mode is not set to tcp, so it has default value ipc, so salt-api is working properly, but it's breaking functionality of external authentication..

root@sm-3007:~# grep ipc /etc/salt/master.d/external_auth.conf
#ipc_mode: tcp
root@sm-3007:~# systemctl restart salt-master.service salt-minion
root@sm-3007:~# ls -la /var/run/salt/master/
total 0
drwxr-xr-x 2 root root 120 Jun 14 10:13 .
drwxr-xr-x 4 root root  80 Jun 14 08:01 ..
srw------- 1 root root   0 Jun 14 10:13 master_event_pub.ipc
srw------- 1 root root   0 Jun 14 10:13 master_event_pull.ipc
srw------- 1 root root   0 Jun 14 10:13 publish_pull.ipc
srw------- 1 root root   0 Jun 14 10:13 workers.ipc

salt-api call using pepper

user@sm-3007:~$ salt-pepper '*' test.version
{
    "return": [
        {
            "sm-3007": "3007.1"
        }
    ]
}

..but run using external authentication module:

user@sm-3007:~$ salt -C '*' test.ping
password:
[WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds -   File "/usr/bin/salt", line 11, in <module>
    sys.exit(salt_main())

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 528, in salt_main
    client.run()

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
    for full_ret in cmd_func(**kwargs):

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 815, in cmd_cli
    self.pub_data = self.run_job(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 387, in run_job
    pub_data = self.pub(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1904, in pub
    if listen and not self.event.connect_pub(timeout=timeout):

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub
    self.subscriber = salt.utils.asynchronous.SyncWrapper(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 76, in __init__
    self.obj = cls(*args, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client
    return publish_client(opts, io_loop, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client
    return salt.transport.tcp.PublishClient(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 220, in __init__
    super().__init__(opts, io_loop, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in __init__
    super().__init__()

[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
TypeError: argument must be an int, or have a fileno() method.
Traceback (most recent call last):
  File "/usr/bin/salt", line 11, in <module>
    sys.exit(salt_main())
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 528, in salt_main
    client.run()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
    for full_ret in cmd_func(**kwargs):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 830, in cmd_cli
    for fn_ret in self.get_cli_event_returns(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1653, in get_cli_event_returns
    for ret in self.get_iter_returns(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1187, in get_iter_returns
    for raw in ret_iter:
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1104, in get_returns_no_block
    raw = self.event.get_event(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 651, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 553, in _get_event
    raw = self.subscriber.recv(timeout=wait)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 138, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 146, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/ioloop.py", line 527, in run_sync
    return future_cell[0].result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 375, in recv
    events, _, _ = select.select([self._stream.socket], [], [], 0)
TypeError: argument must be an int, or have a fileno() method.
Traceback (most recent call last):
  File "/usr/bin/salt", line 11, in <module>
    sys.exit(salt_main())
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 528, in salt_main
    client.run()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
    for full_ret in cmd_func(**kwargs):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 830, in cmd_cli
    for fn_ret in self.get_cli_event_returns(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1653, in get_cli_event_returns
    for ret in self.get_iter_returns(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1187, in get_iter_returns
    for raw in ret_iter:
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1104, in get_returns_no_block
    raw = self.event.get_event(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 651, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 553, in _get_event
    raw = self.subscriber.recv(timeout=wait)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 138, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 146, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/ioloop.py", line 527, in run_sync
    return future_cell[0].result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 375, in recv
    events, _, _ = select.select([self._stream.socket], [], [], 0)
TypeError: argument must be an int, or have a fileno() method.

.. which is pointing to permissions issues.. so after adding rw permissions for others to /var/run/salt/master/master_event_pub.ipc which I am aware that is REALLY STUPID IDEA.. and I wouldn't mention this aloud in decent society.

root@sm-3007:~# chmod o+rw /var/run/salt/master/master_event_pub.ipc
root@sm-3007:~# ls -la /var/run/salt/master/master_event_pub.ipc
srw----rw- 1 root root 0 Jun 14 10:13 /var/run/salt/master/master_event_pub.ipc
root@sm-3007:~# su - user
user@sm-3007:~$ salt -C '*' test.ping
sm-3007:
    True

Expected behavior
I would expect that external authentication against pam or ldap is not mutually exclusive with using salt-api. So there is possibility to use both features at same time.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
root@sm-3007:~# salt --versions-report
Salt Version:
          Salt: 3007.1

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: ubuntu 22.04.4 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-107-generic
        system: Linux
       version: Ubuntu 22.04.4 jammy

Additional context
Run user is set to root because of intend to authenticate locally using pam module.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the configuration using external_auth, ipc_mode: tcp, and rest_cherrypy, then start with the IPC and event paths shown in salt/transport/tcp.py and salt/utils/event.py. Check the salt-api login path and the master socket permissions described in the report. Done means PAM or LDAP external authentication and salt-api both work after a clean restart without manually changing socket permissions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, authentication, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.