saltstack / saltstack/salt

3008.x: warn_until() allocates ~15,000 SaltStackVersion objects/sec under stress

Open Beginner friendly
#69,921 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

salt.utils.versions.warn_until() constructs a fresh salt.version.SaltStackVersion on every call for both the target-version comparison and the running-version comparison:

  • salt/utils/versions.py:171
    _version_ = salt.version.SaltStackVersion(*_version_info_)
    
  • salt/utils/versions.py:143-156 — target-version resolution constructs another SaltStackVersion per call

SaltStackVersion.__init__ builds a packaging.version.Version internally.

Impact

A memray capture on the master's EventPublisher process during 4h stress showed ~1.4 M packaging.version.Version allocations totalling 2.5 GB in a 90 s window. Every call site inside the transport hot path (e.g. TCPPubClient.__init__, TCPReqServer.__init__, MessageClient.__init__ — all deprecated aliases still instantiated during connection setup and dispatch) fires warn_until and therefore allocates 2 SaltStackVersion objects per call — even though the actual warnings.warn(...) message is filtered when PYTHONWARNINGS isn't set.

At ~15,000 calls/sec sustained, the transient allocation churn is ~28 MB/sec. Glibc arena high-water-mark grows to accommodate the peak burst, then never releases — the process's RSS is pinned above its actual working set indefinitely.

Reproduction
import tracemalloc, warnings
warnings.filterwarnings("ignore")
import salt.version, salt.utils.versions

_orig = salt.version.SaltStackVersion.__init__
n = [0]
def counting(self, *a, **kw):
    n[0] += 1
    return _orig(self, *a, **kw)
salt.version.SaltStackVersion.__init__ = counting

n[0] = 0
for _ in range(10_000):
    salt.utils.versions.warn_until(3009, "test")
print(n[0], "SaltStackVersion() calls")   # -> 20000
Proposed fix

Memoize the two SaltStackVersion constructions in warn_until. Target-version resolution has 32 slots (functools.lru_cache(maxsize=32)) which is comfortably larger than the number of distinct constant-arg call sites in the codebase; running-version resolution takes the _version_info_ tuple as key (typically a single value for the process lifetime).

After the patch, the same 10,000-call loop makes 0 SaltStackVersion constructions (100% reduction on the repeated-argument path).

Measured impact on production-shaped stress

30-min WebSocket-transport master test with the patch alone (vs 4h unpatched baseline captured in the same rig):

  • EventPublisher RSS peak: 271 MB → 214 MB (-57 MB / -21%)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/utils/versions.py at the warn_until() target-version and running-version construction points, then inspect the listed transport call sites to understand the hot path. Use the provided counting reproduction to verify repeated calls no longer construct SaltStackVersion objects, and run the relevant versions tests. Done means preserving warning behavior while eliminating repeated allocations for repeated arguments.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.