saltstack / saltstack/salt

[Bug]: cp.cache_file fails for files served from secondary file_roots path after upgrade to 3006.27

Open
#69,811 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

What happened?

After upgrading a Salt minion from 3006.26 to 3006.27, cp.cache_file fails for any file served from the second entry in file_roots. Files from the first file_roots path continue to work. Downgrading the minion back to 3006.26 immediately resolves the issue.

Setup:

Master file_roots configuration:
file_roots:
base:
- /srv/salt/states
- /srv/salt/files

Steps to Reproduce:

Upgrade minion to salt-minion 3006.27 (master remains on 3006.27)
Verify the file is visible to the minion:
salt-call cp.list_master | grep myfile.tar.gz

  • files/packages/myfile.tar.gz # file is listed
    Attempt to cache a file from the second file_roots path:
    salt-call cp.cache_file salt:///files/packages/myfile.tar.gz
    [ERROR] Unable to cache file 'salt:///files/packages/myfile.tar.gz' from saltenv 'base'.
    local: False
    Caching a file from the first file_roots path works fine:
    salt-call cp.cache_file salt://mystate/init.sls
    local: /var/cache/salt/minion/files/base/mystate/init.sls
    Debug output (with -l debug):

[DEBUG] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'MYMINION', 'tcp://10.x.x.x:4506', '1784038030.4247723')
[DEBUG] <salt.crypt.AsyncAuth object at 0x7f4c3ae272d0> Got new master aes key.
[DEBUG] Closing AsyncReqChannel instance
[DEBUG] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'MYMINION', 'tcp://10.x.x.x:4506', '1784038030.4247723')
[DEBUG] Closing AsyncReqChannel instance
[ERROR] Unable to cache file 'salt:///files/packages/myfile.tar.gz' from saltenv 'base'.
On another minion (running 3006.26) in the same environment, the same cp.cache_file command succeeds immediately.

A separate minion also running 3006.27 showed an additional clue in the debug log:

[ERROR] Future <salt.ext.tornado.concurrent.Future object at 0x7f8ec271e5c0> exception was never retrieved:
AttributeError: 'NoneType' object has no attribute 'set_result'
AttributeError: 'NoneType' object has no attribute 'set_exception'

Key observations:
test.ping works (port 4505 communication is fine)
cp.list_master shows the file (the fileserver metadata channel works)
Network connectivity to port 4506 is confirmed (nc -zv succeeds)
The issue affects files of any size (tested with both ~200MB and ~5MB files) from the secondary file_roots path
Files from the primary file_roots path cache without issues
The master log shows no activity when the failing request is made
Fresh minion installation (complete removal + bootstrap reinstall) with 3006.27 still exhibits the bug
Downgrading to 3006.26 immediately fixes the issue

Versions:
Salt master: 3006.27 (Sulfur)
Salt minion (affected): 3006.27 (Sulfur)
Salt minion (working): 3006.26 (Sulfur)
OS: RHEL 8 / CentOS Stream 8

Type of salt install

Official rpm

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

rhel-8

salt --versions-report output
[root@MASTER nikotest]$ salt --versions-report
Salt Version:
          Salt: 3006.27

Python Version:
        Python: 3.11.15 (main, Jun 29 2026, 22:21:49) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 47.0.0
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.50
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.1.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.16
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.5

System Versions:
          dist: rhel 8.10 Ootpa
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.144.1.el8_10.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 8.10 Ootpa


[root@MINION_BUGGY ~]$ salt-call --versions-report
Salt Version:
          Salt: 3006.27

Python Version:
        Python: 3.11.15 (main, Jun 29 2026, 22:21:49) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 47.0.0
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.1.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.16
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.5



System Versions:
          dist: rhel 8.10 Ootpa
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.144.1.el8_10.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 8.10 Ootpa

[root@MINION ~]$ salt-call --versions-report
Salt Version:
          Salt: 3006.26

Python Version:
        Python: 3.10.20 (main, Jun 11 2026, 02:19:33) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 47.0.0
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.1.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.14
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.5

System Versions:
          dist: rhel 8.10 Ootpa
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.144.1.el8_10.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 8.10 Ootpa

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with salt-call cp.cache_file for files in the second file_roots entry, comparing 3006.27 with 3006.26. Trace the cp.cache_file request path alongside cp.list_master and the reported NoneType Future errors, then verify that files from secondary roots cache successfully without affecting primary-root files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.