3008.x: RequestRouter._extract_command does per-request AES decrypt + msgpack decode even when routing table has no explicit mappings

Description

On 3008.x the MWorkerQueue process runs a Python-level dispatch loop (salt/transport/zeromq.py:zmq_device_pooled) that calls RequestRouter.route_request(payload) on every incoming request. route_request in turn calls RequestRouter._extract_command (salt/master.py:1650-1703), which for every AES-encrypted payload:

1. Allocates a fresh salt.crypt.Crypticle(self.opts, key) per call (salt/master.py:1673),
2. Runs AES-CBC + HMAC-SHA256 over the full body to decrypt it,
3. Msgpack-decodes the entire payload dict via salt.payload.loads (salt/master.py:1685) just to read the single load["cmd"] field,
4. Then discards both the Crypticle and the decrypted dict — the MWorker downstream decrypts the same bytes a second time in its _handle_aes -> AESFuncs.run_func path.

For deployments whose worker-pools layout has no per-command mapping (all commands land in the same catchall pool), the routing decision is a constant regardless of cmd. That includes:

• Any master that uses the legacy worker_threads setting -- get_worker_pools_config (salt/config/worker_pools.py:253-261) synthesizes a single pool with commands: [\"*\"], producing an empty cmd_to_pool on the router.

In those configurations 100% of the AES decrypt and 100% of the msgpack-decode work above is dead work.

Impact

Under sustained request rate (`_return` / `_pillar` / `_mine_get` traffic) the transient allocations from per-request Crypticle.__init__, Crypticle.decrypt, and salt.payload.loads on the full payload stack up in pymalloc arenas. Under a 4h stress rig against 3-worker 3008.x master (see tests/monitoring/), the MWorkerQueue process grows RSS by ~380 MB beyond what libzmq's own churn accounts for. A memray-backed audit at agents/reports/zmq-master-app-leak-audit.md identifies this call site as the top suspect (rank #1 and #3 in that report).

Root cause

_extract_command unconditionally attempts the decrypt/decode when it sees encrypted bytes, even though the routing decision does not depend on cmd when self.cmd_to_pool is empty. self.cmd_to_pool is populated once at RequestRouter.__init__ from worker_pools and never modified afterwards (salt/master.py:1579-1607), so the check is stable for the process lifetime.

Proposed fix

Short-circuit _extract_command at the top when self.cmd_to_pool is empty:

def _extract_command(self, payload):
    if not self.cmd_to_pool:
        # Single-pool deployment: every command routes to self.default_pool.
        # Skip the AES decrypt + msgpack decode entirely.
        return \"\"
    ...

Because _classify_request returns self.cmd_to_pool.get(cmd, self.default_pool), returning \"\" falls through to the catchall on line 1648. No routing behaviour changes for the single-pool case; only the dead work disappears. Deployments that explicitly configure per-command pool mappings continue to hit the decrypt/decode path unchanged.

AES key rotation continues to propagate through self.secrets.get(\"aes\", {}).get(\"secret\", {}).value on the multi-pool path -- the short-circuit does not touch that reference.

Reproduction

Stress rig at tests/monitoring/ (already checked in). The 4h numbers above are reproducible on a 3-worker master.

Start in salt/master.py at RequestRouter._extract_command and trace route_request through salt/transport/zeromq.py:zmq_device_pooled. Review the single-pool configuration in salt/config/worker_pools.py and run the reproduction under tests/monitoring/. Done means single-pool requests avoid the redundant work while explicitly mapped commands retain existing routing and decryption behavior.