saltstack / saltstack/salt

--static flag causes longer run time

Open
#56,180 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Confirmed Performance severity-high
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

When running a salt command with the --static flag it causes the run time of a module function to increases by order of magnitude.
eg.
Example function targeting approximately 1000 minions:

time sudo salt --out=json -C * module.module_func

<removed_sensitive data>
.
.
real	0m12.666s
user	0m2.512s
sys	0m0.380s

vs when adding the `--static flag:

time sudo salt --static--out=json -C * module.module_func

<removed_sensitive data>
.
.
real	1m36.855s
user	0m3.166s
sys	0m0.405s

Forcing a low timeout does not help the static case either, I am trying to use the --static flag to get valid/parseable JSON as a whole from the output.

This issue has arisen recently without any changes to our salt deployment other than adding in a new reactor that listens for high states:

reactor:
  - 'salt/job/*/new':
    - salt://_reactor/console-highstate.sls

console-highstate.sls:

{% if data['fun'] == 'state.highstate' %}
changed_name_for_senistive_data:
  runner.runner_name.runner_fun:
    - args:
      - target: {{ data['minions'] | json }}
{% endif %}

The current work around is to not use the --static flag and perform string manipulation to make the whole output valid JSON. Would like to know why the static flag is causing this large increase in time though. None of the minions are failing from the call in either case where the flag is set or not.

Versions Report
           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: 2.0.5
      gitpython: 2.1.14
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: 1.3.10
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15+ (default, Jul  9 2019, 16:51:35)
   python-gnupg: 0.4.1
         PyYAML: 3.13
          PyZMQ: 18.1.0
           RAET: Not Installed
          smmap: 2.0.5
        timelib: Not Installed
        Tornado: 5.1.1
            ZMQ: 4.3.2

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-1032-aws
         system: Linux
        version: Ubuntu 18.04 bionic```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported salt command against a large minion target with and without --static, comparing the runtimes and JSON output. Start from the CLI handling of --static and trace why it waits longer; done means identifying and correcting the slowdown while preserving valid whole-output JSON.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.