[BUG] Master/API/Minion resource leaks under sustained load (process, FD, memory)

Description

Under sustained load on 3008.x (thousands of events/sec, rapid-fire netapi logins, highstate loops), the master, salt-api, and minion processes leak processes, file descriptors, and memory. Master RSS balloons past 7 GB and eventually OOMs. Investigation on 3006.x reproduced the same root causes; they are present in 3008.x head.

Filing as one tracking issue for the whole set — each sub-item has an independent fix.

Reproduction

Stress harness in tests/monitoring/ (docker-compose master + minions + salt-api + Prometheus/Grafana). Drivers:

• flood_events.py — fire salt events at ~5,000/sec into the event bus.
• stress_test.sh — parallel state.apply / test.ping / manage.status loops.
• stress_api.sh — CherryPy netapi login + client=local / client=runner in a tight loop.

Observed within minutes: MWorker RSS climbs unbounded, master FD count climbs, zombie children accumulate. Within an hour master hits multi-GB RSS.

Leak inventory (root causes identified)

Process / FD leaks

1. salt/utils/timed_subprocess.py — TimedProc.kill() does not wait(), orphaning zombies to PID 1 after timeout.
2. salt/utils/event.py — SaltEvent/MasterEvent have no context-manager protocol and no full teardown; ZMQ sockets and event-bus connections outlive their callers.
3. salt/netapi/__init__.py — NetapiClient instantiates LocalClient / RunnerClient / WheelClient / SSHClient per request without teardown. Primary source of the CherryPy TCP/RAM leak.
4. salt/runner.py, salt/wheel/__init__.py — no destroy(); callers using bare instantiation leak per-call.

Memory leaks / cache bloat

5. salt/loader/lazy.py — LazyLoader never purges _dict, loaded_modules, or sys.modules entries. Every reload leaks the module set → "module creep" on both master and minion.
6. salt/utils/context.py, salt/utils/ctx.py — ContextDict / RequestContext.current_request not cleared on teardown; per-request state pinned indefinitely.
7. salt/master.py, salt/daemons/masterapi.py — AESFuncs / RemoteFuncs bound methods (e.g. _serve_file) create refcycles that pin Fileserver and friends past worker recycle.
8. salt/utils/args.py — get_function_argspec() defined _ArgSpec = namedtuple(...) at function scope, dynamically compiling a fresh class (C-level struct) on every module call. At high call rates this is a fast, unbounded leak.
9. Worker recycling + PyZMQ asyncio.Task: closing ZMQStream sockets mid-flight while _run_callback has wrapped coroutines via asyncio.ensure_future permanently leaks the Task objects into the running loop. Aggressive worker_resource_backcount recycling made this worse, not better.
10. Destructor cascades missing on the loader-owning classes — Fileserver, Cache, Roster, MasterMinion, RemoteFuncs/LocalFuncs, NetapiClient, LoadAuth, Runner. Even when callers do the right thing, the owned LazyLoaders stay in sys.modules.

Scalability

11. salt/tokens/localfs.py — list_tokens uses os.walk and returns a full list; on token dirs with 1M+ entries this is a GB-scale spike during clean_expired_tokens. Same code path in salt/daemons/masterapi.py Maintenance.

IPC

12. EventPublisher inherits Tornado's default ipc_write_buffer (~27 GB effective). A slow consumer causes unbounded RAM growth on the publisher instead of a disconnect.

Fix status (staged locally on 3006.x branch dwoz/3006leak; forward-porting to 3008.x)

• TimedProc.wait() after kill().
• SaltEvent/MasterEvent context managers + full destroy().
• NetapiClient uses with for all sub-clients; Runner/Wheel gain destroy() and context-manager use.
• LazyLoader.destroy() explicitly purges _dict, loaded_modules, and matching sys.modules keys.
• ContextDict.destroy(); RequestContext clears current_request.
• AESFuncs.destroy() / RemoteFuncs.destroy() nullify bound-method attributes (self._serve_file = None, etc.) to break the refcycle.
• _ArgSpec namedtuple hoisted to module scope in salt/utils/args.py.
• worker_resource_backcount recycle path removed — workers cache execution context; saves ~600 MB globally and eliminates the PyZMQ Task leak. (If a bounded reset is still wanted, it must first drain outstanding ZMQStream callbacks.)
• Destructor cascades wired into Fileserver, Cache, Roster, MasterMinion, RemoteFuncs, LocalFuncs, NetapiClient, LoadAuth, Runner.
• list_tokens converted to os.scandir-based generator; Maintenance.clean_expired_tokens consumes the generator (constant memory).
• Recommend a sane default cap for ipc_write_buffer on EventPublisher (or hard cap + disconnect on overflow).

Post-fix validation

• Master baseline ~70 MB / peak ~155 MB RSS (was 7 GB+ before OOM).
• Minion ~108 MB stable (was 1.3 GB).
• salt-api ~53 MB stable.
• Master FDs stable at ~1,300 under ~5,700 events/sec.

Remaining slow leak (unresolved)

After the above, MWorker still creeps ~25 MB/hr under sustained stress; MWorkerQueue occasionally accumulates. Profiler shows ReferenceType, Context, generator, and Task counts ticking upward. Suspects (unconfirmed):

• Residual salt.ext.tornado.zmq.eventloop.zmqstream future retention.
• Unbounded cache inside a hot execution module (test.ping, manage.status, grains.items) or in CherryPy session handling.
• IOLoop.add_future callbacks not clearing on external-auth paths.

Will file a follow-up narrowly-scoped issue once tracemalloc/objgraph isolate it.

Target

3008.x. 3006.x carries the same code paths (verified) but is LTS — separate backport scope.

salt --versions-report

3008.x head (branch tip at time of filing). Reproduction environment is Ubuntu 24.04 containers under docker-compose (see tests/monitoring/).

Start with the stress harness in tests/monitoring/ and choose one independently scoped leak item, then read its named entry point such as salt/utils/timed_subprocess.py, salt/utils/event.py, or salt/loader/lazy.py. Run the relevant stress driver and add or update focused tests where the subsystem provides them. Done means the selected resource remains bounded under the stated workload and the harness reproduces the reported validation.