saltstack / saltstack/salt

[BUG] Synchronous batch releases slots while jobs are still running

Open
#70,018 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

Synchronous CLI batching in Salt 3008.x can publish the next sub-batch after
timeout + gather_job_timeout has elapsed even when the current minion's job
is still running. This allows actual concurrency to exceed the requested batch
size.

Mechanism

The shared batch state machine applies its internal wall-clock timeout sweep to
both driver="master" and driver="cli". For the synchronous CLI driver,
however, LocalClient.cmd_iter_no_block() is already authoritative for job
liveness: its return iterator probes saltutil.find_job and remains alive
while the job is running.

The internal sweep bypasses that tracking. It removes the minion from active,
marks it timed out, and opens a slot even though the iterator is still tracking
the running job.

Expected behavior

For a synchronous CLI batch, a slot should remain active until the iterator
returns the minion result or exhausts and reports the minion timed out. The
master-driven asynchronous batch still needs its internal wall-clock timeout
because it has no LocalClient return iterator.

Regression coverage

Two tests reproduce the problem with synthetic minions:

  1. A state-machine test advances the clock beyond the timeout window and
    verifies that a CLI slot remains occupied until timed_out is explicitly
    reported.
  2. A Batch.run() test keeps the first return iterator alive past that window
    and verifies that the second sub-batch is not published before the first
    return arrives.

Both tests fail with the current 3008.x logic and pass when the internal sweep
is restricted to driver="master".

This appears to reintroduce the behavior reported in #8597 and fixed by #11184
during the async batching refactor in #68964.

Version

Observed and reproduced against 3008.2 / the current 3008.x branch.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the shared batch state machine and Batch.run(), then trace how LocalClient.cmd_iter_no_block() reports job liveness. Run the two described regression tests: the state-machine timeout test and the Batch.run() iterator test. Done means CLI slots remain occupied until the iterator returns or reports a timeout, while master-driven asynchronous batching still performs its internal sweep.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.