saltstack / saltstack/salt

[BUG] runner sometimes not available in gitfs

Open
#68,051 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
I'm trying to create a salt runner. My salt configuration is stored in a self-hosted git instance.
Sometimes it works after pushing the runner, sometimes it isn't recognized anymore.

Setup

root@salt:~# salt-run saltutil.sync_all 
cache:
clouds:
engines:
executors:
fileserver:
grains:
modules:
output:
pillar:
proxymodules:
queues:
renderers:
returners:
roster:
runners:
sdb:
serializers:
states:
thorium:
tokens:
tops:
utils:
wheel:
root@salt:~# ls -l /var/cache/salt/master/extmods/runners/
total 8
-rw------- 1 salt salt 6094 Jun  3 12:58 hostkeygen.py
root@salt:~# salt-run saltutil.sync_all -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/autosign.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/autosign.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/gpg.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/gpg.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/nodegroups.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/nodegroups.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/pillar.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/pillar.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/source.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/source.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt.mgmt.sci.hpi.de
[DEBUG   ] Missing configuration file: /root/.saltrc
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] The functions from module 'saltutil' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded saltutil.sync_all
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/autosign.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/autosign.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/gpg.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/gpg.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/nodegroups.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/nodegroups.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/pillar.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/pillar.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/source.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/source.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt.mgmt.sci.hpi.de
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/autosign.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/autosign.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/gpg.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/gpg.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/nodegroups.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/nodegroups.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/pillar.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/pillar.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/source.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/source.conf
[DEBUG   ] The functions from module 'core' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'disks' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'extra' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'lvm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'mdadm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from '/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/zfs.py'>
[DEBUG   ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded zfs.is_supported
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Popen(['git', 'version'], cwd=None, stdin=None, shell=False, universal_newlines=False)
[DEBUG   ] Popen(['git', 'version'], cwd=None, stdin=None, shell=False, universal_newlines=False)
[DEBUG   ] The functions from module 'hostkeygen' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'auth' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'cache' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'cloud' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'config' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'doc' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'error' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'event' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'fileserver' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'git_pillar' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'http' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'jobs' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'launchd' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'lxc' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'manage' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'mattermost' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'mine' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'network' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'pagerduty' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'pillar' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'pkg' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'queue' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'reactor' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'salt' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'sdb' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'vmadm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'ssh' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'survey' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'test' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'thin' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'vault' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'virt' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'winrepo' are being loaded by dir() on the loaded module
[DEBUG   ] Sending event: tag = salt/run/20250603130408961193/new; data = {'fun': 'runner.saltutil.sync_all', 'jid': '20250603130408961193', 'user': 'salt', 'fun_args': [], '_stamp': '2025-06-03T13:04:09.688682'}
[DEBUG   ] Syncing all
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/clouds'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Current fetch URL for gitfs remote 'git@gitlab.hpi.de:sci/configuration/salt-stack.git': git@gitlab.hpi.de:sci/configuration/salt-stack.git (desired: git@gitlab.hpi.de:sci/configuration/salt-stack.git)
[DEBUG   ] Current refspecs for gitfs remote 'git@gitlab.hpi.de:sci/configuration/salt-stack.git': ['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*'])
[DEBUG   ] Current http.sslVerify for gitfs remote 'git@gitlab.hpi.de:sci/configuration/salt-stack.git': true (desired: true)
[DEBUG   ] register_cleanup_finalize_function entry, function=<function gitfs_finalize_cleanup at 0x7259d7302560>, args=('/var/cache/salt/master',), kwargs={}
[DEBUG   ] Created gitfs object for process 301005
[INFO    ] Syncing clouds for environment 'base'
[INFO    ] Loading cache from salt://_clouds, for base
[INFO    ] Caching directory '_clouds/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[PROFILE ] gitfs file_name cache rebuild repo=git@gitlab.hpi.de:sci/configuration/salt-stack.git duration=0.03995347023010254 seconds
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_clouds'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/modules'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing modules for environment 'base'
[INFO    ] Loading cache from salt://_modules, for base
[INFO    ] Caching directory '_modules/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_modules'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/states'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing states for environment 'base'
[INFO    ] Loading cache from salt://_states, for base
[INFO    ] Caching directory '_states/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_states'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/grains'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing grains for environment 'base'
[INFO    ] Loading cache from salt://_grains, for base
[INFO    ] Caching directory '_grains/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_grains'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/renderers'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing renderers for environment 'base'
[INFO    ] Loading cache from salt://_renderers, for base
[INFO    ] Caching directory '_renderers/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_renderers'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/returners'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing returners for environment 'base'
[INFO    ] Loading cache from salt://_returners, for base
[INFO    ] Caching directory '_returners/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_returners'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/output'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing output for environment 'base'
[INFO    ] Loading cache from salt://_output, for base
[INFO    ] Caching directory '_output/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_output'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/proxy'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing proxy for environment 'base'
[INFO    ] Loading cache from salt://_proxy, for base
[INFO    ] Caching directory '_proxy/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_proxy'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing runners for environment 'base'
[INFO    ] Loading cache from salt://_runners, for base
[INFO    ] Caching directory '_runners/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] In saltenv 'base', looking at rel_path '_runners/hostkeygen.py' to resolve 'salt://_runners/hostkeygen.py'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/master/files/base/_runners/hostkeygen.py' to resolve 'salt://_runners/hostkeygen.py'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://_runners/hostkeygen.py'
[DEBUG   ] No dest file found
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Fetching file from saltenv 'base', ** done ** '_runners/hostkeygen.py'
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_runners'
[INFO    ] Copying '/var/cache/salt/master/files/base/_runners/hostkeygen.py' to '/var/cache/salt/master/extmods/runners/hostkeygen.py'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/wheel'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing wheel for environment 'base'
[INFO    ] Loading cache from salt://_wheel, for base
[INFO    ] Caching directory '_wheel/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_wheel'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/engines'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing engines for environment 'base'
[INFO    ] Loading cache from salt://_engines, for base
[INFO    ] Caching directory '_engines/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_engines'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/thorium'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing thorium for environment 'base'
[INFO    ] Loading cache from salt://_thorium, for base
[INFO    ] Caching directory '_thorium/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_thorium'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/queues'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing queues for environment 'base'
[INFO    ] Loading cache from salt://_queues, for base
[INFO    ] Caching directory '_queues/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_queues'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/pillar'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing pillar for environment 'base'
[INFO    ] Loading cache from salt://_pillar, for base
[INFO    ] Caching directory '_pillar/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_pillar'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/utils'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing utils for environment 'base'
[INFO    ] Loading cache from salt://_utils, for base
[INFO    ] Caching directory '_utils/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_utils'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/sdb'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing sdb for environment 'base'
[INFO    ] Loading cache from salt://_sdb, for base
[INFO    ] Caching directory '_sdb/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_sdb'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/cache'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing cache for environment 'base'
[INFO    ] Loading cache from salt://_cache, for base
[INFO    ] Caching directory '_cache/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_cache'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/fileserver'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing fileserver for environment 'base'
[INFO    ] Loading cache from salt://_fileserver, for base
[INFO    ] Caching directory '_fileserver/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_fileserver'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/tops'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing tops for environment 'base'
[INFO    ] Loading cache from salt://_tops, for base
[INFO    ] Caching directory '_tops/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_tops'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/tokens'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing tokens for environment 'base'
[INFO    ] Loading cache from salt://_tokens, for base
[INFO    ] Caching directory '_tokens/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_tokens'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/serializers'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing serializers for environment 'base'
[INFO    ] Loading cache from salt://_serializers, for base
[INFO    ] Caching directory '_serializers/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_serializers'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/executors'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing executors for environment 'base'
[INFO    ] Loading cache from salt://_executors, for base
[INFO    ] Caching directory '_executors/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_executors'
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/roster'
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Created gitfs object with uninitialized remotes
[DEBUG   ] The functions from module 'gitfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded gitfs.envs
[DEBUG   ] Re-using gitfs object for process 301005
[INFO    ] Syncing roster for environment 'base'
[INFO    ] Loading cache from salt://_roster, for base
[INFO    ] Caching directory '_roster/' for environment 'base'
[DEBUG   ] Re-using gitfs object for process 301005
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/master/file_lists/gitfs/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_roster'
[DEBUG   ] The functions from module 'local_cache' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded local_cache.prep_jid
[DEBUG   ] Adding minions for job 20250603130408961193: []
[DEBUG   ] Sending event: tag = salt/run/20250603130408961193/ret; data = {'fun': 'runner.saltutil.sync_all', 'jid': '20250603130408961193', 'user': 'salt', 'fun_args': [], '_stamp': '2025-06-03T13:04:10.314808', 'return': {'clouds': [], 'modules': [], 'states': [], 'grains': [], 'renderers': [], 'returners': [], 'output': [], 'proxymodules': [], 'runners': [], 'wheel': [], 'engines': [], 'thorium': [], 'queues': [], 'pillar': [], 'utils': [], 'sdb': [], 'cache': [], 'fileserver': [], 'tops': [], 'tokens': [], 'serializers': [], 'executors': [], 'roster': []}, 'success': True}
[DEBUG   ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded nested.output
cache:
clouds:
engines:
executors:
fileserver:
grains:
modules:
output:
pillar:
proxymodules:
queues:
renderers:
returners:
roster:
runners:
sdb:
serializers:
states:
thorium:
tokens:
tops:
utils:
wheel:
[INFO    ] Runner completed: 20250603130408961193
[DEBUG   ] Closing IPCMessageClient instance
[DEBUG   ] Runner return: {'fun': 'runner.saltutil.sync_all', 'jid': '20250603130408961193', 'user': 'salt', 'fun_args': [], '_stamp': '2025-06-03T13:04:10.314808', 'return': {'clouds': [], 'modules': [], 'states': [], 'grains': [], 'renderers': [], 'returners': [], 'output': [], 'proxymodules': [], 'runners': [], 'wheel': [], 'engines': [], 'thorium': [], 'queues': [], 'pillar': [], 'utils': [], 'sdb': [], 'cache': [], 'fileserver': [], 'tops': [], 'tokens': [], 'serializers': [], 'executors': [], 'roster': []}, 'success': True}
root@salt:~# salt-run runners.hostkeygen "test.node.cloud.sci.hpi.de"
'runners.hostkeygen' is not available.

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify) QEMU in Proxmox
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
I don't really know, I have just added the runner in the top of the repository in the _runners folder.

Expected behavior
The runner is always recognized

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
salt --versions-report
Salt Version:
          Salt: 3006.10
 
Python Version:
        Python: 3.10.16 (main, Mar  6 2025, 02:23:15) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.17.1
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.44
        Jinja2: 3.1.6
       libgit2: 1.9.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.17.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.18.1
         smmap: 5.0.2
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 24.04.2 noble
        locale: utf-8
       machine: x86_64
       release: 6.8.0-60-generic
        system: Linux
       version: Ubuntu 24.04.2 noble

Additional context

This is the salt master config:

root@salt:~# cat /etc/salt/master.d/*
autosign_file: /etc/salt/autosign.conf
gpg_keydir: /etc/salt/gpgkeys
nodegroups:
  hpc: 'L@*.hpc.domain.com'
  cloud: 'L@*.cloud.domain.com'
  mgmt: 'L@*.mgmt.domain.com'
ext_pillar:
  - git:
      - __env__ git@gitlab.domain.com:group/configuration/salt-stack.git:
          - root: pillar          # repo folder that contains pillar data
          - privkey: /mykey
          - pubkey: /mykey.pub

# base → production for pillar as well
git_pillar_base: production
reactor:
  - 'salt/auth':
      - salt://_reactors/hostkey_manage.sls
fileserver_backend:
  - gitfs

gitfs_provider: pygit2
gitfs_pubkey: /mykey.pub
gitfs_privkey: /mykey
gitfs_update_interval: 300

gitfs_remotes:
  - git@gitlab.domain.com:group/configuration/salt-stack.git

# map base → production
gitfs_base: production

state_top_saltenv: base

If anything else is required, please let me know. :)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the /etc/salt/master configuration and the salt-run saltutil.sync_all entry point, then follow the gitfs provider messages and the /var/cache/salt/master/extmods/runners/ cache shown in the report. Reproduce the intermittent sync after pushing a runner and determine why the runner is not consistently recognized; done means the runner is available reliably after synchronization.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.