saltstack / saltstack/salt

[BUG] salt-api server dies on bad /login request

Open
#62,187 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Salt-API
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Salt-master times out when doing salt-api /auth endpoint request without username or password fields but with eauth field present. Can lead to salt-api DDoS.

Setup
Really the most generic salt setup you can ever create, just enabled salt-api with external_auth (config below)

  • VM KVM Master and slave

Steps to Reproduce the behavior

  • Install salt-master, salt-minion and salt-api packages
  • Enable external_auth in master config
Example master config 
external_auth:
  auto:
    admin:
      - .*

rest_cherrypy:
  port: 8000
  disable_ssl: True
- Try to do salt-api `/login` request without password field
Example curl 
curl -X "POST" "http://{{salt_endpoint}}:8000/login" -H 'Content-Type: application/json; charset=utf-8' -d '{"eauth": "auto",  "username": "admin","token": "whatever"}'
- See salt-api getting stuck, master errors in logs, retries in salt-api (every retry takes 1 minute - why is this default?)
Salt-api stuck logs 
2022-06-17 10:20:11,965 [salt.loaded.int.netapi.rest_cherrypy.app:747 ][INFO    ][3027991] [api_acl] Authentication not checked for user admin from IP 192.168.41.101
2022-06-17 10:20:11,974 [salt.transport.zeromq:158 ][DEBUG   ][3027991] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
2022-06-17 10:20:11,976 [salt.transport.zeromq:1233][DEBUG   ][3027991] Trying to connect to: tcp://127.0.0.1:4506
2022-06-17 10:21:12,036 [salt.transport.zeromq:1291][DEBUG   ][3027991] SaltReqTimeoutError, retrying. (1/3)
2022-06-17 10:22:12,094 [salt.transport.zeromq:1291][DEBUG   ][3027991] SaltReqTimeoutError, retrying. (2/3)
2022-06-17 10:23:12,156 [salt.transport.zeromq:1291][DEBUG   ][3027991] SaltReqTimeoutError, retrying. (3/3)
2022-06-17 10:24:12,212 [salt.transport.zeromq:1261][DEBUG   ][3027991] Re-init ZMQ socket: Message timed out
2022-06-17 10:24:12,214 [salt.transport.zeromq:1233][DEBUG   ][3027991] Trying to connect to: tcp://127.0.0.1:4506
2022-06-17 10:24:12,216 [salt.transport.zeromq:179 ][DEBUG   ][3027991] Closing AsyncZeroMQReqChannel instance
2022-06-17 10:24:12,218 [cherrypy.access.140230875023728:284 ][INFO    ][3027991] 192.168.41.101 - - [17/Jun/2022:10:24:12] "POST /login HTTP/1.1" 503 742 "" "curl/7.79.1"
Salt-master logs 
022-06-17 10:20:09,059 [salt.config      :1917][DEBUG   ][3027939] Reading configuration from /etc/salt/master
2022-06-17 10:20:09,062 [salt.config      :2080][DEBUG   ][3027939] Including configuration from '/etc/salt/minion.d/_schedule.conf'
2022-06-17 10:20:09,062 [salt.config      :1917][DEBUG   ][3027939] Reading configuration from /etc/salt/minion.d/_schedule.conf
2022-06-17 10:20:09,063 [salt.config      :2080][DEBUG   ][3027939] Including configuration from '/etc/salt/minion.d/local_master.conf'
2022-06-17 10:20:09,063 [salt.config      :1917][DEBUG   ][3027939] Reading configuration from /etc/salt/minion.d/local_master.conf
2022-06-17 10:20:09,064 [salt.config      :3520][DEBUG   ][3027939] Using cached minion ID from /etc/salt/minion_id: vm-3d-salt-test.pine.ly
2022-06-17 10:20:09,066 [salt.loader      :868 ][DEBUG   ][3027939] Grains refresh requested. Refreshing grains.
2022-06-17 10:20:09,066 [salt.config      :1917][DEBUG   ][3027939] Reading configuration from /etc/salt/master
2022-06-17 10:20:09,086 [salt.utils.entrypoints:57  ][DEBUG   ][3027939] Using importlib_metadata to load entry points
2022-06-17 10:20:09,141 [salt.loader.lazy :857 ][DEBUG   ][3027939] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from '/usr/lib/python3/dist-packages/salt/grains/zfs.py'>
2022-06-17 10:20:09,170 [salt.modules.network:2132][DEBUG   ][3027939] Elapsed time getting FQDNs: 0.017315387725830078 seconds
2022-06-17 10:20:09,335 [salt.utils.lazy  :99  ][DEBUG   ][3027939] LazyLoaded zfs.is_supported
2022-06-17 10:20:09,398 [salt.utils.lazy  :99  ][DEBUG   ][3027939] LazyLoaded local_cache.clean_old_jobs
2022-06-17 10:20:09,435 [salt.utils.lazy  :99  ][DEBUG   ][3027939] LazyLoaded localfs.list_tokens
2022-06-17 10:20:09,437 [salt.utils.verify:454 ][DEBUG   ][3027939] This salt-master instance has accepted 1 minion keys.
2022-06-17 10:20:11,981 [salt.utils.lazy  :99  ][DEBUG   ][3027950] LazyLoaded auto.auth
2022-06-17 10:20:11,982 [salt.transport.zeromq:795 ][ERROR   ][3027950] Some exception handling a payload from minion
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/transport/zeromq.py", line 791, in handle_message
    ret, req_opts = yield self.payload_handler(payload)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 294, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3.8/types.py", line 278, in wrapped
    coro = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/master.py", line 1023, in _handle_payload
    ret = {"aes": self._handle_aes, "clear": self._handle_clear}[key](load)
  File "/usr/lib/python3/dist-packages/salt/master.py", line 1064, in _handle_clear
    ret = method(load), {"fun": "send_clear"}
  File "/usr/lib/python3/dist-packages/salt/master.py", line 2130, in mk_token
    token = self.loadauth.mk_token(clear_load)
  File "/usr/lib/python3/dist-packages/salt/auth/__init__.py", line 209, in mk_token
    if not self.authenticate_eauth(load):
  File "/usr/lib/python3/dist-packages/salt/auth/__init__.py", line 312, in authenticate_eauth
    if not self.time_auth(load):
  File "/usr/lib/python3/dist-packages/salt/auth/__init__.py", line 115, in time_auth
    ret = self.__auth_call(load)
  File "/usr/lib/python3/dist-packages/salt/auth/__init__.py", line 98, in __auth_call
    fcall = salt.utils.args.format_call(
  File "/usr/lib/python3/dist-packages/salt/utils/args.py", line 470, in format_call
    raise SaltInvocationError(
salt.exceptions.SaltInvocationError: auth takes at least 2 arguments (1 given)
2022-06-17 10:20:11,990 [tornado.general  :467 ][ERROR   ][3027950] Uncaught exception in zmqstream callback
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/zmq/sugar/socket.py", line 435, in send_multipart
    memoryview(msg)
TypeError: memoryview: a bytes-like object is required, not 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 460, in _handle_events
    self._handle_send()
  File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 499, in _handle_send
    status = self.socket.send_multipart(msg, **kwargs)
  File "/usr/lib/python3/dist-packages/zmq/sugar/socket.py", line 440, in send_multipart
    raise TypeError(
TypeError: Frame 0 ('Some exception handling minion ...) does not support the buffer interface.
2022-06-17 10:20:11,994 [tornado.application:640 ][ERROR   ][3027950] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f8617aeb5e0>)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/zmq/sugar/socket.py", line 435, in send_multipart
    memoryview(msg)
TypeError: memoryview: a bytes-like object is required, not 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
    ret = callback()
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 542, in <lambda>
    self.io_loop.add_callback(lambda : self._handle_events(self.socket, 0))
  File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 460, in _handle_events
    self._handle_send()
  File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 499, in _handle_send
    status = self.socket.send_multipart(msg, **kwargs)
  File "/usr/lib/python3/dist-packages/zmq/sugar/socket.py", line 440, in send_multipart
    raise TypeError(
TypeError: Frame 0 ('Some exception handling minion ...) does not support the buffer interface.
- All further requests to salt-api via rest_cherrypy salt-api server become stuck - Further bogus requests prolong server downtime - If we stop doing bad requests server will come back

Expected behavior

  • Do bad request to salt-api (/login without password field)
  • See bad request error and not connection getting stuck

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
Salt Version:
          Salt: 3004.1
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: 0.28.3
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: 1.0.3
        Python: 3.8.10 (default, Mar 15 2022, 12:22:08)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2
 
System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.13.0-41-generic
        system: Linux
       version: Ubuntu 20.04 focal

Additional context
Problem can be easily solved by placing try block before format_call() func here: https://github.com/saltstack/salt/blob/master/salt/auth/__init__.py#L98-L101

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/auth/init.py around __auth_call and the format_call() call, then reproduce the malformed /login request with the curl example. Verify that missing authentication fields return a bad-request error without blocking salt-api or causing later requests to get stuck.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, security
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.