saltstack / saltstack/salt

[BUG] pkg.installed with sources re-downloads every time

Open
#59,241 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug pending-discussion severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Using pkg.installed on Debian with sources to install a .deb fetches over https results in re-downloading the same file every run.

Setup
Reproducible with an empty minion config and basic SLS (as /tmp/install.sls):

sops_package:
  pkg.installed:
    - sources:
      - sops: https://github.com/mozilla/sops/releases/download/v3.6.1/sops_3.6.1_amd64.deb

Steps to Reproduce the behavior

The sops package is already installed:

# salt-call --local pkg.version sops
local:
    3.6.1

And the file is already in the cache:

 # ls -l /var/cache/salt/minion/extrn_files/github.com/mozilla/sops/releases/download/v3.6.1/sops_3.6.1_amd64.deb 
-rw-r--r-- 1 root root 13135134 Jan  6 16:09 /var/cache/salt/minion/extrn_files/github.com/mozilla/sops/releases/download/v3.6.1/sops_3.6.1_amd64.deb

But running the state causes a re-download:

# salt-call --local --log-level debug state.template /tmp/install.sls
[...]
[INFO    ] Running state [sops_package] at time 16:38:19.150264
[INFO    ] Executing state pkg.installed for [sops_package]
[INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/home/satwell'
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Requesting URL https://github.com/mozilla/sops/releases/download/v3.6.1/sops_3.6.1_amd64.deb using GET method
[DEBUG   ] Using backend: tornado
[INFO    ] Executing command ['dpkg', '-I', '/var/cache/salt/minion/extrn_files/github.com/mozilla/sops/releases/download/v3.6.1/sops_3.6.1_amd64.deb'] in directory '/home/satwell'
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [sops_package] at time 16:38:23.480707 (duration_in_ms=4330.451)
[DEBUG   ] File /var/cache/salt/minion/accumulator/140577000434176 does not exist, no need to cleanup
[DEBUG   ] LazyLoaded highstate.output
local:
----------
          ID: sops_package
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 16:38:19.150256
    Duration: 4330.451 ms
     Changes:   

Summary for local
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1 
Total run time:   4.330 s

Expected behavior

Because I've specified the name of a package that's already installed, I wouldn't expect the state do anything. But even if it does make sense to check the package name/version against the specified .deb file, I'd expect salt to use the file already in the cache rather than re-downloading.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:        
          Salt: 3002.2                                                                         
  
Dependency Versions:
          cffi: 1.14.4                                                                         
      cherrypy: Not Installed                                                                  
      dateutil: 2.8.1                          
     docker-py: 4.1.0                          
         gitdb: Not Installed                                                                  
     gitpython: Not Installed                                                                  
        Jinja2: 2.11.2                                                                         
       libgit2: Not Installed                                                                  
      M2Crypto: Not Installed                                                                  
          Mako: 1.1.3
       msgpack: 1.0.0                          
  msgpack-pure: Not Installed                  
  mysql-python: Not Installed                  
     pycparser: 2.20                           
      pycrypto: 2.6.1                          
  pycryptodome: 3.9.7
        pygit2: Not Installed                                                                  
        Python: 3.9.1 (default, Dec  8 2020, 07:51:42)                                         
  python-gnupg: 0.4.6                          
        PyYAML: 5.3.1
         PyZMQ: 20.0.0 
         smmap: Not Installed                  
       timelib: Not Installed                  
       Tornado: 4.5.3
           ZMQ: 4.3.3
            
System Versions:                               
          dist: debian testing bullseye
        locale: utf-8                          
       machine: x86_64                         
       release: 5.9.0-4-amd64      
        system: Linux
       version: Debian GNU/Linux testing bullseye                                              
  

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the pkg.installed entry point and the /tmp/install.sls reproduction on Debian, then trace how a sources URL is resolved against the minion cache. Verify repeated runs do not request the cached .deb again while preserving the already-installed result, and add or update coverage for this behavior if the relevant tests are located.

Written by the indexing model from the issue text.

Assessment

Tech stack
debian, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.