[BUG] Minion returns randomly missing in batch mode
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
Sometimes some minion returns are missing in the output of a batch mode command.
For a command like salt -C 'I@salt:minion' --out=json -b'70%' cmd.run "salt-call cmd.run 'echo OK'" the output might look like:
....<<skip>>....
{
"jid": "20200919150026255687",
"retcode": 0,
"node-11": "[INFO ] Executing command 'echo OK' in directory '/root'\nlocal:\n OK"
}
{
"jid": "20200919150027708040",
"node-12": "[INFO ] Executing command 'echo OK' in directory '/root'\nlocal:\n OK",
"retcode": 0
}
{
"node-6": {}
}
Although all minions are alive, in some cases some outputs are missing. The minions are not overloaded and there are no networking issues. Oftentimes, in subsequent runs the same minion outputs are missing. When the issue occurs, a "find_job" job with the target of the missing minion appears in jobs.list_jobs:
20200919162151938541:
----------
Arguments:
- 20200919162051760278
Function:
saltutil.find_job
StartTime:
2020, Sep 19 16:21:51.938541
Target:
- node-6
Target-type:
list
User:
salt-user
Setup
This particular setup has 23 minions but the configuration is tuned for much bigger environments.
master.conf:
sock_pool_size: 15
zmq_backlog: 3000
gather_job_timeout: 30
worker_threads: 20
timeout: 60
max_open_files: 15000
state_output: changes
pillar_opts: False
pillar_safe_render_error: True
auto_accept: True
max_event_size: 100000000
minion.conf:
master: 10.0.0.1
id: node-6
acceptance_wait_time: 10
acceptance_wait_time_max: 60
auth_timeout: 180
master_tries: -1
max_event_size: 100000000
random_reauth_delay: 270
recon_default: 1000
recon_max: 60000
recon_randomize: True
Steps to Reproduce the behavior
It's enough to run salt -C 'I@salt:minion' --out=json -b'70%' cmd.run "salt-call cmd.run 'echo OK'" several times.
On my setup with 23 minions it gives almost 50% reproduction rate.
Expected behavior
All returns from all minions are present in the output.
Versions Report
Initially caught on 2017.7.x series, reproduced also on 3000.3.
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3000.3
Dependency Versions:
cffi: 1.9.1
cherrypy: 3.5.0
dateutil: 2.4.2
docker-py: 1.9.0
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.8
libgit2: Not Installed
M2Crypto: 0.21.1
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.12 (default, Nov 12 2018, 14:36:49)
python-gnupg: 0.3.8
PyYAML: 3.11
PyZMQ: 15.2.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.15.0-43-generic
system: Linux
version: Ubuntu 16.04 xenial
Additional context
Only the batch client is affected, does not reproduce with any other client.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.