saltstack / saltstack/salt

[BUG] Connection timeout errors from salt-netapi-client since upgrading salt-master 3005.3 to 3006.4

Open
#65,641 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Since upgrading salt-master to 3006.4, our salt-api java client (salt-netapi-client) has been failing with connection timeout errors:

java.util.concurrent.CompletionException: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-491 [ACTIVE]
  at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
  at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
  at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
  at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
  at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
  at com.suse.salt.netapi.client.impl.HttpAsyncClientImpl$1.failed(HttpAsyncClientImpl.java:104)

Running state.apply with pepper python client, I also see frequent severe slow requests, although no connection timeout errors.

e.g.

> time pepper -v -t 20 -p test --client local_async <minionid> state.apply formulas.1_0_0-1686614879.succeeds3
{
    "return": [
        {
            "jid": "20231129010943569061",
            "minions": [
                "<minionid>"
            ]
        }
    ]
}
pepper -v -t 20 -p test --client local_async <minionid>  0.16s user 0.04s system 35% cpu 0.573 total

vs

> time pepper -v -t 20 -p test --client local_async <minionid> state.apply formulas.1_0_0-1686614879.succeeds3
{
    "return": [
        {
            "jid": "20231129011009422677",
            "minions": [
                "<minionid>"
            ]
        }
    ]
}
pepper -v -t 20 -p test --client local_async <minionid>  0.16s user 0.04s system 0% cpu 25.464 total

These erratic slow responses have only shown up when we started testing salt-master 3006.4 (previously using 3005.3)

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify) docker
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

Use pepper client to launch async jobs using --client local_async and any formula. Note that periodically the pepper call take >15s

TODO: I will work on a simple java test doing the same, but this behavior with pepper I think may be close enough to be similar cause.

Expected behavior

async jobs should not take more than 1s to launch using the api.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
(root) ~ # salt --versions
Salt Version:
          Salt: 3006.4
 
Python Version:
        Python: 3.10.13 (main, Oct  4 2023, 21:54:22) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: 1.1.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.6.1
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.12
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 22.04.3 jammy
        locale: utf-8
       machine: x86_64
       release: 6.5.10netflix-g1c6990acc598
        system: Linux
       version: Ubuntu 22.04.3 jammy

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the delay with the pepper --client local_async command against Salt 3006.4 in the documented Docker and onedir setup, then compare it with 3005.3. Check the salt-api request path and the Java HttpAsyncClientImpl timeout reported in the stack trace. Done means identifying the 3006.4 regression and confirming async jobs launch within the expected one-second target without client timeouts.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java, python
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.