[BUG] Unrealistic 'duration' Values in Salt Jobs for state.highstate Operations
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
While running the state.highstate function for managing server configuration, I noticed unusually high duration values in the SaltStack logs. Below are two examples of such logs. The reported durations are over 86,000 seconds, which seems incorrect.
Setup
- on-prem machine
- VM KVM
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
Run the salt-call state.highstate command on the affected minion.
Check the logs in /var/log/salt/minion.
Observe the reported duration values. Note that the issue does not occur consistently and seems to happen randomly.
2025-01-21 12:08:10,457 [salt.state :2312][INFO ][2482] Running state [/etc/pki/rpm-gpg/RPM-GPG-KEY-elasticsearch] at time 12:08:10.457217 2025-01-21 12:08:10,457 [salt.state :2343][INFO ][2482] Executing state file.managed for [/etc/pki/rpm-gpg/RPM-GPG-KEY-elasticsearch] 2025-01-21 12:08:10,166 [salt.state :320 ][INFO ][2482] File /etc/pki/rpm-gpg/RPM-GPG-KEY-elasticsearch is in the correct state 2025-01-21 12:08:10,167 [salt.state :2510][INFO ][2482] Completed state [/etc/pki/rpm-gpg/RPM-GPG-KEY-elasticsearch] at time 12:08:10.166998 (**duration_in_ms=86399709.781**)
2025-01-22 12:08:03,815 [salt.state :2312][INFO ][2486] Running state [/etc/backup/client.conf] at time 12:08:03.814996 2025-01-22 12:08:03,815 [salt.state :2343][INFO ][2486] Executing state file.managed for [/etc/backup/client.conf] 2025-01-22 12:08:03,627 [salt.state :320 ][INFO ][2486] File /etc/backup/client.conf is in the correct state 2025-01-22 12:08:03,628 [salt.state :2510][INFO ][2486] Completed state [/etc/backup/client.conf] at time 12:08:03.627972 (**duration_in_ms=86399812.975**)
Expected behavior
The duration values should reflect the actual time taken to execute the tasks. Values such as 86399709.781, 86399812.975 are implausibly high and do not match the operations performed.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)# salt-minion --versions-report
Salt Version:
Salt: 3006.9
Python Version:
Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
cryptography: 42.0.5
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.4
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.17.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: rocky 8.10 Green Obsidian
locale: utf-8
machine: x86_64
release: 4.18.0-553.34.1.el8_10.x86_64
system: Linux
version: Rocky Linux 8.10 Green Obsidian
Additional context
Logs show duration values that are far too high for the described operations.
Please advise on whether this is a known issue or provide guidance on further debugging steps.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing salt-call state.highstate on the reported Rocky Linux setup and inspect /var/log/salt/minion around the state execution timestamps. Compare the logged durations with the actual operation time and determine why intermittent values near 86,400 seconds occur; done means the reported durations match the work performed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100