saltstack / saltstack/salt

[BUG] Unrealistic 'duration' Values in Salt Jobs for state.highstate Operations

Open
#67,189 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

While running the state.highstate function for managing server configuration, I noticed unusually high duration values in the SaltStack logs. Below are two examples of such logs. The reported durations are over 86,000 seconds, which seems incorrect.

Setup

  • on-prem machine
  • VM KVM
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior

Run the salt-call state.highstate command on the affected minion.
Check the logs in /var/log/salt/minion.
Observe the reported duration values. Note that the issue does not occur consistently and seems to happen randomly.

2025-01-21 12:08:10,457 [salt.state :2312][INFO ][2482] Running state [/etc/pki/rpm-gpg/RPM-GPG-KEY-elasticsearch] at time 12:08:10.457217 2025-01-21 12:08:10,457 [salt.state :2343][INFO ][2482] Executing state file.managed for [/etc/pki/rpm-gpg/RPM-GPG-KEY-elasticsearch] 2025-01-21 12:08:10,166 [salt.state :320 ][INFO ][2482] File /etc/pki/rpm-gpg/RPM-GPG-KEY-elasticsearch is in the correct state 2025-01-21 12:08:10,167 [salt.state :2510][INFO ][2482] Completed state [/etc/pki/rpm-gpg/RPM-GPG-KEY-elasticsearch] at time 12:08:10.166998 (**duration_in_ms=86399709.781**)

2025-01-22 12:08:03,815 [salt.state :2312][INFO ][2486] Running state [/etc/backup/client.conf] at time 12:08:03.814996 2025-01-22 12:08:03,815 [salt.state :2343][INFO ][2486] Executing state file.managed for [/etc/backup/client.conf] 2025-01-22 12:08:03,627 [salt.state :320 ][INFO ][2486] File /etc/backup/client.conf is in the correct state 2025-01-22 12:08:03,628 [salt.state :2510][INFO ][2486] Completed state [/etc/backup/client.conf] at time 12:08:03.627972 (**duration_in_ms=86399812.975**)

Expected behavior

The duration values should reflect the actual time taken to execute the tasks. Values such as 86399709.781, 86399812.975 are implausibly high and do not match the operations performed.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
# salt-minion --versions-report
Salt Version:
          Salt: 3006.9
 
Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: rocky 8.10 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.34.1.el8_10.x86_64
        system: Linux
       version: Rocky Linux 8.10 Green Obsidian

Additional context

Logs show duration values that are far too high for the described operations.
Please advise on whether this is a known issue or provide guidance on further debugging steps.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing salt-call state.highstate on the reported Rocky Linux setup and inspect /var/log/salt/minion around the state execution timestamps. Compare the logged durations with the actual operation time and determine why intermittent values near 86,400 seconds occur; done means the reported durations match the work performed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.