3008.x: RequestRouter builds a fresh Crypticle per AES request, driving pymalloc arena churn in MWorkerQueue

Description

On 3008.x with the default pooled request path (worker_pools_enabled: True), the MWorkerQueue process runs a Python-level dispatch loop that calls RequestRouter._extract_command on every request. For AES-encrypted payloads (the common case: minion _return, _pillar, _mine_get, _register_resources) it builds a fresh Crypticle per request:

• salt/master.py:1668-1674

  if enc == "aes":
      key = self.secrets.get("aes", {}).get("secret", {}).value
      if key:
          import salt.crypt
          crypticle = salt.crypt.Crypticle(self.opts, key)  # every request
          load = crypticle.decrypt(load)
  

• salt/crypt.py:2005-2008

  @classmethod
  def extract_keys(cls, key_string, key_size):
      key = salt.utils.stringutils.to_bytes(base64.b64decode(key_string))
      assert len(key) == key_size / 8 + cls.SIG_SIZE, "invalid key"
      return key[: -cls.SIG_SIZE], key[-cls.SIG_SIZE :]
  

Every AES request triggers Crypticle.__init__ -> extract_keys -> base64.b64decode(...) (fresh bytes) plus AES/HMAC key derivation. Under sustained ~10-50 req/s (typical busy master with minion returns) this drives pymalloc arena high-water climb; extrapolated to 200-350 MB / 4h RSS growth in MWorkerQueue.

Impact

Same stress rig used for #69920 (3 worker pools, flood_events + state.apply + state.highstate + salt-api curl loop). The suspect was identified during the leak audit under agents/reports/zmq-master-app-leak-audit.md (top-ranked suspect, estimated 200-350 MB of the ~380 MB gap).

Root cause

RequestRouter._extract_command needs the AES key only to inspect load["cmd"] for pool routing. The Crypticle is discarded immediately after decrypt, so every request re-derives AES/HMAC subkeys from the same shared-memory secret. SMaster.secrets["aes"]["secret"] is a multiprocessing.Array(ctypes.c_char, ...) shared across master processes; the value changes only on key rotation (SMaster.rotate_secrets), which mutates the shared buffer in place (.value = new_bytes).

Note: multiprocessing.Array.value returns a fresh bytes object on every read, so caching by id(secret.value) would rebuild on every request and defeat the purpose. Caching must key on the bytes value itself.

Proposed fix

Cache the Crypticle on the RequestRouter instance, keyed by the current AES key bytes; rebuild only when the shared-memory value changes (i.e. on rotation). Bytes comparison per request is ~200x cheaper than full Crypticle construction.

# salt/master.py:RequestRouter
def __init__(self, opts, secrets=None):
    ...
    self._crypticle = None
    self._crypticle_key_bytes = None  # last key bytes we cached against

def _get_crypticle(self):
    secret = self.secrets.get("aes", {}).get("secret")
    key_bytes = getattr(secret, "value", None)
    if not key_bytes:
        return None
    if self._crypticle_key_bytes != key_bytes:
        import salt.crypt
        self._crypticle = salt.crypt.Crypticle(self.opts, key_bytes)
        self._crypticle_key_bytes = key_bytes
    return self._crypticle

Rotation via SMaster.rotate_secrets mutates the shared buffer, next .value read observes the new bytes, and the cache rebuilds transparently. Per-request cost is one bytes comparison; the AES-CBC + HMAC key derivation runs once per key generation, not once per request.

This is the belt-and-braces companion to a separate short-circuit that skips _extract_command decrypt entirely on single-pool deployments (where cmd_to_pool is empty and everything lands in the catchall). The Crypticle cache handles the multi-pool case where decryption is genuinely needed on every request.

Reproduction

Stress rig at tests/monitoring/ (already checked in); allocations attributable to Crypticle.__init__ / extract_keys visible under tracemalloc.

Start in salt/master.py at RequestRouter._extract_command and the Crypticle construction path, then run the stress rig in tests/monitoring with tracemalloc. Verify that repeated AES requests reuse the cached object, that key rotation causes it to refresh, and that the reported allocation growth is reduced without changing pool routing.