[BUG] salt-api 401 Unauthorized

Description
Since upgrading two of our Saltstack environments we seem to face a bug of the salt-api that refuses any requests.
The problem occurs on both systems, although I am only giving details of one environment so far. But both are pretty similar.
The API worked like expected before.

As you can see I am running the OneDir version 3006.4 as root. Previously we were running version 3006.3 and had a lot of problems with the NonceVerificationError

The login function works well and returns a token and also shows the right permissions.
curl -k http://salt:8080/login -H "Accept:application/json" -H "Content-type:application/json" -d '{"username":"tux", "password":"areallystrongpassword", "eauth":"pam"}' {"return": [{"token": "areallysecuretoken", "expire": 1700522599.8706264, "start": 1700479399.8706262, "user": "tux", "eauth": "pam", "perms": [".*", "@jobs", "@runner", "@wheel"]}]}

But the following request fails.
curl -k http://salt:8080/ -H "Content-type:application/json" -H "X-Auth-Token:areallysecuretoken" -d '[{"client":"wheel","fun":"key.list"}]'
No matter what fun and client I use and also when using "eauth=pam" with the same credentials. The result is always the same:

<h2>401 Unauthorized</h2>
        <p>No permission -- see authorization schemes</p>
        <pre id="traceback">Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 638, in respond
    self._do_respond(path_info)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 688, in _do_respond
    self.hooks.run('before_request_body')
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 95, in run
    self.run_hooks(iter(sorted(self[point])))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 117, in run_hooks
    hook()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 65, in __call__
    return self.callback(**self.kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/netapi/rest_cherrypy/app.py", line 781, in salt_auth_tool
    raise cherrypy.HTTPError(401)
cherrypy._cperror.HTTPError: (401, None)
</pre>
    <div id="powered_by">
      <span>
        Powered by <a href="http://www.cherrypy.org">CherryPy 18.6.1</a>
      </span>
    </div>

Here's what's in the log:

Nov 20 12:31:34 salt salt-api[23062]: 10.8.25.129 - - [20/Nov/2023:12:31:34] "POST /login HTTP/1.1" 200 208 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0"
Nov 20 12:31:48 salt salt-master[22446]: [WARNING ] Authentication failure of type "token" occurred.
Nov 20 12:31:48 salt salt-api[23062]: [ERROR   ] Authorization error occurred.
Nov 20 12:31:48 salt salt-api[23062]: 10.8.25.129 - - [20/Nov/2023:12:31:48] "POST / HTTP/1.1" 401 1036 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0"

Seems to me like somehow the master is the problem here that refuses the login.

On the CLI the user is able to run the commands:

root@salt:/etc/salt# salt -a pam --username=tux sdev-salt01 test.ping
password:
sdev-salt01:
    True

Hopefully I am not wasting your time with a pretty stupid layer 8 problem.
Thanks a lot!

ADDITION:
I am still looking for a solution and found out a few additional things:

• The following request runs as part of on of our gitlab pipelines and it still works:
  curl -sS http://$SALTAPI/hook/$CI_PROJECT_NAME/commit -H "Content-type:application/json" -H "X-Auth-Token:$TOKEN" -d '{"user-mail":"$GITLAB_USER_EMAIL", "user-name":"$GITLAB_USER_NAME"}' | grep success
• This request works too:
  curl http://salt02:8080/run -H 'Accept: application/x-yaml' -d username=tux -d password=securepassword -d eauth=pam -d client=local -d tgt='*' -d fun=test.ping

So, right now it looks to me like there is a problem with all the requests on the root path of the webserver.

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (VMware)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

salt-api config:

rest_cherrypy:
  host: 0.0.0.0
  port: 8080
  debug: True
  disable_ssl: True
  log_access_file: /var/log/salt/api-access.log
  log_error_file: /var/log/salt/api-error.log

netapi_enable_clients:
  - local
  - local_async
  - local_batch
  - local_subset
  - runner
  - runner_async
  - wheel
  - wheel_async

external_auth:
  pam:
    tux:
      - '.*'
      - '@runner'
      - '@jobs'
      - '@wheel'

Master Config:

root@salt:/opt# cat /etc/salt/master | grep -vE '^#' | grep -vE '^$'
user: root
event_return: mysql
keep_acl_in_token: True

Here are some more logs of the master on'debug' level:

Nov 20 13:06:19 salt salt-master[46022]: [DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
Nov 20 13:06:23 salt salt-master[46014]: [DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
Nov 20 13:06:26 salt salt-master[46025]: [DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
Nov 20 13:06:27 salt salt-master[46024]: [DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
Nov 20 13:06:28 salt salt-master[46003]: [WARNING ] Authentication failure of type "token" occurred.
Nov 20 13:06:28 salt salt-master[46003]: [DEBUG   ] Sending event: tag = salt/job/error; data = {'cmd': 'publish', 'tgt': 'sdev-salt02', 'fun': 'test.ping', 'arg': [], 'key': 'somekey', 'tgt_type': 'glob', 'ret': '', 'jid': '', 'kwargs': {'client': 'local', 'token': 'reallysecuretoken'}, 'user': 'root', 'error': {'name': 'AuthorizationError', 'message': 'Authorization error occurred.'}, '_stamp': '2023-11-20T12:06:28.238982'}
Nov 20 13:06:28 salt salt-master[45997]: [DEBUG   ] Flushing 1 events.
Nov 20 13:06:28 salt salt-master[45997]: [DEBUG   ] Calling event returner mysql, only one configured.
Nov 20 13:06:28 salt salt-master[45958]: [DEBUG   ] Gathering reactors for tag salt/job/error
Nov 20 13:06:28 salt salt-master[45997]: [DEBUG   ] Using default for mysql ssl_ca
Nov 20 13:06:28 salt salt-master[45997]: [DEBUG   ] Using default for mysql ssl_cert
Nov 20 13:06:28 salt salt-master[45997]: [DEBUG   ] Using default for mysql ssl_key
Nov 20 13:06:28 salt salt-master[45997]: [DEBUG   ] Trying to reuse MySQL connection pool
Nov 20 13:06:28 salt salt-api[36960]: [ERROR   ] Authorization error occurred.
Nov 20 13:06:28 salt salt-api[36960]: 10.8.25.129 - - [20/Nov/2023:13:06:28] "POST / HTTP/1.1" 401 1036 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0"

Steps to Reproduce the behavior

Expected behavior
Please just do what I want :)

Screenshots

Versions Report

Additional context

Start in salt/netapi/rest_cherrypy/app.py at the salt_auth_tool handler mentioned in the traceback. Reproduce the authenticated / request with the supplied curl command, then compare it with the working /run and /hook requests and review the token-related master logs. Done means authenticated root-path requests succeed consistently while the existing /run and /hook behavior remains intact.