[BUG] CherryPy Could not serialize the return data f... | saltstack/salt

Nobody has claimed this yet.

Description
When attempting to access the /stats endpoint via the salt-api, I receive the following response:

<!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
    <title>500 Internal Server Error</title>
    <style type="text/css">
    #powered_by {
        margin-top: 20px;
        border-top: 2px solid black;
        font-style: italic;
    }

    #traceback {
        color: red;
    }
    </style>
</head>
    <body>
        <h2>500 Internal Server Error</h2>
        <p>Could not serialize the return data from Salt.</p>
        <pre id="traceback"></pre>
    <div id="powered_by">
      <span>
        Powered by <a href="http://www.cherrypy.org">CherryPy 8.9.1</a>
      </span>
    </div>
    </body>
</html>

Setup

rest_cherrypy:
  collect_stats: True
  port: 7101
  disable_ssl: True
  static: /var/www/salt-api
  static_path: /healthcheck

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

Follow the instructions here: https://docs.saltproject.io/en/latest/ref/netapi/all/salt.netapi.rest_cherrypy.html#authentication to get an authentication token for the api.

curl -sS http://localhost:7101/login -H 'Accept: application/json' -d username=saltdev -d password=saltdev -d eauth=pam
{"return": [{"token": "xxx", "expire": 1614346756.7356985, "start": 1614303556.7356982, "user": "saltdev", "eauth": "pam", "perms": [".*", "@jobs", "@runner"]}]}

Use the token to access the stats endpoint.

# curl -sS http://localhost:7101/stats -H 'Accept: application/json' -H 'X-Auth-Token: xxx'
<!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
    <title>405 Method Not Allowed</title>
    <style type="text/css">
    #powered_by {
        margin-top: 20px;
        border-top: 2px solid black;
        font-style: italic;
    }

    #traceback {
        color: red;
    }
    </style>
</head>
    <body>
        <h2>405 Method Not Allowed</h2>
        <p>Specified method is invalid for this resource</p>
        <pre id="traceback"></pre>
    <div id="powered_by">
      <span>
        Powered by <a href="http://www.cherrypy.org">CherryPy 8.9.1</a>
      </span>
    </div>
    </body>
</html>

Make the same request, but for the yaml output, and you receive the stats.

curl -sS http://localhost:7101/stats -H 'Accept: application/x-yaml' -H 'X-Auth-Token: xxx'
CherryPy Applications:
  Bytes Read/Request: 2.028225806451613
  Bytes Read/Second: 0.21173904402536514
  Bytes Written/Request: 8986.592741935483
  Bytes Written/Second: 938.1660302839282
  Current Requests: 1
  Current Time: 1614303967.352832
  Enabled: true
  Requests: .... << truncated as it's really large

/var/log/salt/api has no information on the error.

Expected behavior
I expect the same data returned when requesting YAML to be returned when requesting JSON.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

# salt -V
Salt Version:
          Salt: 3002.5
 
Dependency Versions:
          cffi: 1.11.5
      cherrypy: unknown  << Shows as Unknown but the version from the installed package is 8.9.1-2
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: 2.0.3
     gitpython: 2.1.8
        Jinja2: 2.10
       libgit2: 0.26.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.18
      pycrypto: 2.6.1
  pycryptodome: 3.4.7
        pygit2: 0.26.2
        Python: 3.6.9 (default, Jan 26 2021, 15:33:00)
  python-gnupg: 0.4.1
        PyYAML: 3.12
         PyZMQ: 17.1.2
         smmap: 2.0.3
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.5
 
System Versions:
          dist: ubuntu 18.04 Bionic Beaver
        locale: UTF-8
       machine: x86_64
       release: 5.4.0-1037-aws
        system: Linux
       version: Ubuntu 18.04 Bionic Beaver

Additional context
Happy to produce any additional information that might help debug this issue.

Start at the salt-api REST CherryPy /stats endpoint and the rest_cherrypy collect_stats configuration, reproducing the issue with the provided JSON and YAML curl requests. Compare the two content-negotiation paths and trace why JSON serialization fails; done means JSON returns the same stats data as YAML. The payload names no source files or tests.

Written by the indexing model from the issue text.

[BUG] CherryPy Could not serialize the return data from Salt

Description

Contributor guide

First steps

Research direction

Assessment