[BUG] Connection timeout errors from salt-netapi-client since upgrading salt-master 3005.3 to 3006.4
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
Since upgrading salt-master to 3006.4, our salt-api java client (salt-netapi-client) has been failing with connection timeout errors:
java.util.concurrent.CompletionException: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-491 [ACTIVE]
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at com.suse.salt.netapi.client.impl.HttpAsyncClientImpl$1.failed(HttpAsyncClientImpl.java:104)
Running state.apply with pepper python client, I also see frequent severe slow requests, although no connection timeout errors.
e.g.
> time pepper -v -t 20 -p test --client local_async <minionid> state.apply formulas.1_0_0-1686614879.succeeds3
{
"return": [
{
"jid": "20231129010943569061",
"minions": [
"<minionid>"
]
}
]
}
pepper -v -t 20 -p test --client local_async <minionid> 0.16s user 0.04s system 35% cpu 0.573 total
vs
> time pepper -v -t 20 -p test --client local_async <minionid> state.apply formulas.1_0_0-1686614879.succeeds3
{
"return": [
{
"jid": "20231129011009422677",
"minions": [
"<minionid>"
]
}
]
}
pepper -v -t 20 -p test --client local_async <minionid> 0.16s user 0.04s system 0% cpu 25.464 total
These erratic slow responses have only shown up when we started testing salt-master 3006.4 (previously using 3005.3)
Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Please be as specific as possible and give set-up details.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify) docker
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
(Include debug logs if possible and relevant)
Use pepper client to launch async jobs using --client local_async and any formula. Note that periodically the pepper call take >15s
TODO: I will work on a simple java test doing the same, but this behavior with pepper I think may be close enough to be similar cause.
Expected behavior
async jobs should not take more than 1s to launch using the api.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)(root) ~ # salt --versions
Salt Version:
Salt: 3006.4
Python Version:
Python: 3.10.13 (main, Oct 4 2023, 21:54:22) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: 1.1.0
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: 1.6.1
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.13.12
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 22.04.3 jammy
locale: utf-8
machine: x86_64
release: 6.5.10netflix-g1c6990acc598
system: Linux
version: Ubuntu 22.04.3 jammy
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the delay with the pepper --client local_async command against Salt 3006.4 in the documented Docker and onedir setup, then compare it with 3005.3. Check the salt-api request path and the Java HttpAsyncClientImpl timeout reported in the stack trace. Done means identifying the 3006.4 regression and confirming async jobs launch within the expected one-second target without client timeouts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java, python
- Domain
- api, backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100