[BUG] Synchronous batch releases slots while jobs are still running
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
Synchronous CLI batching in Salt 3008.x can publish the next sub-batch after
timeout + gather_job_timeout has elapsed even when the current minion's job
is still running. This allows actual concurrency to exceed the requested batch
size.
Mechanism
The shared batch state machine applies its internal wall-clock timeout sweep to
both driver="master" and driver="cli". For the synchronous CLI driver,
however, LocalClient.cmd_iter_no_block() is already authoritative for job
liveness: its return iterator probes saltutil.find_job and remains alive
while the job is running.
The internal sweep bypasses that tracking. It removes the minion from active,
marks it timed out, and opens a slot even though the iterator is still tracking
the running job.
Expected behavior
For a synchronous CLI batch, a slot should remain active until the iterator
returns the minion result or exhausts and reports the minion timed out. The
master-driven asynchronous batch still needs its internal wall-clock timeout
because it has no LocalClient return iterator.
Regression coverage
Two tests reproduce the problem with synthetic minions:
- A state-machine test advances the clock beyond the timeout window and
verifies that a CLI slot remains occupied untiltimed_outis explicitly
reported. - A
Batch.run()test keeps the first return iterator alive past that window
and verifies that the second sub-batch is not published before the first
return arrives.
Both tests fail with the current 3008.x logic and pass when the internal sweep
is restricted to driver="master".
This appears to reintroduce the behavior reported in #8597 and fixed by #11184
during the async batching refactor in #68964.
Version
Observed and reproduced against 3008.2 / the current 3008.x branch.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the shared batch state machine and Batch.run(), then trace how LocalClient.cmd_iter_no_block() reports job liveness. Run the two described regression tests: the state-machine timeout test and the Batch.run() iterator test. Done means CLI slots remain occupied until the iterator returns or reports a timeout, while master-driven asynchronous batching still performs its internal sweep.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100