salt-api can't bear a lot of concurrency when using tornado backend?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
here is question, we want use salt-api to send some shell command to minions, we use tornado_rest salt-api, we have one master and about 800 minions, while we use requests and multiprocessing/threading to make requests to salt-api, when we set about 100 minions and do test.ping function, we get about ten returners like [{}], and if we set about 400 minions, we get many [{}],we use request like this
res = requests.post(
#url=salt_api_url,
url=salt_api_url,
data=json.dumps([{
"client": "local",
"tgt": target,
"fun": "test.ping",
"tgt_type": tgt_type,
"timeout": 10
}]),
headers={
"Accept": "application/json",
"X-Auth-Token": token,
"Content-Type": "application/json"
},
verify=False,
timeout=300
)
and we just want get results synchronously, don't want use async/returners/event,so is there something I don't know with config? can salt-api do as we want? and anyone has some ideas, please leave you message, thanks for all of you.
Setup
saltmaster config
default_include: master.d/*.conf
timeout: 20
worker_threads: 20
auto_accept: True
file_roots:
base:
- /datasalt/srv/salt/base
dev:
- /datasalt/srv/salt/dev
test:
- /datasalt/srv/salt/test
prod:
- /datasalt/srv/salt/prod
pillar_roots:
base:
- /datasalt/srv/pillar/base
dev:
- /datasalt/srv/pillar/dev
test:
- /datasalt/srv/pillar/test
prod:
- /datasalt/srv/pillar/prod
pillar_opts: True
log_level_logfile: debug
salt-api config
rest_tornado:
port: 8001
address: 0.0.0.0
#backlog: 128
ssl_crt: /etc/pki/tls/certs/saltcert.crt
ssl_key: /etc/pki/tls/private/saltcert.key
debug: True
disable_ssl: False
cors_origin: null
webhook_url: /hook
webhook_disable_auth: True
#num_processes: 4
Steps to Reproduce Issue
some logs like this:
2020-01-24 14:24:12,337 [salt.transport.ipc:254 ][DEBUG ][4850] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
2020-01-24 14:24:12,340 [salt.transport.zeromq:1084][DEBUG ][4850] SaltReqTimeoutError, retrying. (1/3)
2020-01-24 14:24:12,380 [salt.transport.zeromq:1084][DEBUG ][4850] SaltReqTimeoutError, retrying. (1/3)
2020-01-24 14:24:12,411 [salt.transport.zeromq:138 ][DEBUG ][4850] Re-using AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'sy-centos7.3.1611-xxxxx_master', u'tcp://1
27.0.0.1:4506', u'clear')
2020-01-24 14:24:12,412 [salt.transport.ipc:254 ][DEBUG ][4850] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
it looks like something wrong with SaltReqTimeoutError, we don't have much error like this with cherrypy_rest salt-api,cherrypy_rest is good but not fast we when have many minions to control at once
Versions Report
Salt Version:
Salt: 2018.3.3
Dependency Versions:
cffi: 1.6.0
cherrypy: unknown
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: 0.24.6
libnacl: Not Installed
M2Crypto: 0.21.1
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: 1.2.5
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.24.2
Python: 2.7.5 (default, Aug 4 2017, 00:39:18)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: centos 7.3.1611 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-693.5.2.el7.x86_64
system: Linux
version: CentOS Linux 7.3.1611 Core
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the salt-api tornado_rest entry point and the supplied master and rest_tornado configuration; reproduce the synchronous requests.post workload at roughly 100 and 400 minions. Trace the SaltReqTimeoutError log path and compare the behavior with cherrypy_rest. Done means the concurrency limit or failure mode is identified and synchronous responses are made reliable or the required configuration is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100