saltstack / saltstack/salt

[Bug]: apt doesn't detect installed package with native architecture in name in install list

Open
#70,209 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

What happened?

This is a followup to #68932

Hey there,

with 3006.26 installed on our nodes, salt's apt handling of packages with the architecture in the name improved. However, there seems to be (at least) 1 bug remaining.
We have this state:

nvidia_driver_bundle:
  pkg.installed:
    - pkgs: {{ driver_pkgs }}
    - require:
      - cmd: update_grub_nouveau
      - pkg: nvidia_repo_keys
      - pkg: install_helper_packages
    - update_holds: True
    - install_recommends: False

With this package list:

  "general": [
    {"nvidia-driver-pinning-{}".format(nvidia_upstream_version): nvidia_full_version},
    {"libnvidia-common": nvidia_full_version},
    {"libnvidia-cfg1:{}".format(arch): nvidia_full_version},
    {"libnvidia-compute:{}".format(arch): nvidia_full_version},
    {"libnvidia-decode:{}".format(arch): nvidia_full_version},
    {"libnvidia-encode:{}".format(arch): nvidia_full_version},
    {"libnvidia-extra:{}".format(arch): nvidia_full_version},
    {"libnvidia-gpucomp:{}".format(arch): nvidia_full_version},
    {"libnvidia-nscq": nvidia_full_version},
    {"nvidia-kernel-common": nvidia_full_version},
    {"nvidia-firmware": nvidia_full_version},
    {"nvidia-modprobe": nvidia_full_version},
    {"nvidia-persistenced": nvidia_full_version},
    "libnvidia-container-tools",
    "libnvidia-container1:{}".format(arch),
  ],

The driver was installed intially successfully. However, now, we see this behaviour:

salt-call --local pkg.install pkgs='["libnvidia-container-tools", "libnvidia-container1:amd64"]' update_holds=True 
[ERROR   ] Command 'systemd-run' failed with return code: 100
[ERROR   ] stdout: Reading package lists...
Building dependency tree...
Reading state information...
The following held packages will be changed:
  libnvidia-container1
The following packages will be upgraded:
  libnvidia-container-tools libnvidia-container1
2 upgraded, 0 newly installed, 0 to remove and 216 not upgraded.
[ERROR   ] stderr: Running as unit: run-r96f57abddd7e4cbcb238653e9dbd5b0d.scope; invocation ID: c7b8fc72ba014b79aa08bae737ae38aa
E: Held packages were changed and -y was used without --allow-change-held-packages.
[ERROR   ] retcode: 100
Error running 'pkg.install': Problem encountered installing package(s). Additional info follows:

changes:
    ----------
errors:
    - Running as unit: run-r96f57abddd7e4cbcb238653e9dbd5b0d.scope; invocation ID: c7b8fc72ba014b79aa08bae737ae38aa
      E: Held packages were changed and -y was used without --allow-change-held-packages.

With dpkg reporting:

dpkg -l | grep nvidia
hi  libnvidia-cfg1:amd64                        590.48.01-0ubuntu1                      amd64        NVIDIA binary OpenGL/GLX configuration library
hi  libnvidia-common                            590.48.01-0ubuntu1                      all          Shared files used by the NVIDIA libraries
hi  libnvidia-compute:amd64                     590.48.01-0ubuntu1                      amd64        NVIDIA libcompute package
hi  libnvidia-container-tools                   1.19.1-1                                amd64        NVIDIA container runtime library (command-line tools)
hi  libnvidia-container1:amd64                  1.19.1-1                                amd64        NVIDIA container runtime library
hi  libnvidia-decode:amd64                      590.48.01-0ubuntu1                      amd64        NVIDIA Video Decoding runtime libraries
hi  libnvidia-encode:amd64                      590.48.01-0ubuntu1                      amd64        NVENC Video Encoding runtime library
hi  libnvidia-extra:amd64                       590.48.01-0ubuntu1                      amd64        Extra libraries for the NVIDIA driver
hi  libnvidia-gpucomp:amd64                     590.48.01-0ubuntu1                      amd64        NVIDIA binary GPU compiler library
hi  libnvidia-nscq                              590.48.01-0ubuntu1                      amd64        NVSwitch Configuration and Query library
hi  nvidia-dkms-open                            590.48.01-0ubuntu1                      amd64        NVIDIA DKMS package (open kernel module)
hi  nvidia-driver-pinning-590.48.01             590.48.01-0ubuntu1                      all          APT driver pinning file for driver version 590.48.01
hi  nvidia-firmware                             590.48.01-0ubuntu1                      amd64        Firmware files used by the kernel module
hi  nvidia-kernel-common                        590.48.01-0ubuntu1                      amd64        Shared files used with the kernel module
hi  nvidia-kernel-source-open                   590.48.01-0ubuntu1                      amd64        NVIDIA kernel source package
hi  nvidia-modprobe                             590.48.01-0ubuntu1                      amd64        Load the NVIDIA kernel driver and create device files
hi  nvidia-persistenced                         590.48.01-0ubuntu1                      amd64        daemon to maintain persistent software state in the NVIDIA driver
ii  nvidia-repo-keys                            26.02-1                                 all          Adds keys to apt trusted.gpg database

Both packages are already installed, but salt thinks that libnvidia-container1 is not already installed, so it makes a call to install it with the newest version, but this doesn't work because it requires the other package to be updated, but it is held because of the state above. We would expect instead that salt does nothing because both packages are already installed.
Interestingly, this state runs through:

salt-call --local pkg.install pkgs='["libnvidia-container-tools", "libnvidia-container1:amd64"]' update_holds=True split_arch=False
local:
    ----------
    libnvidia-container-tools:
        ----------
        new:
            1.20.0-1
        old:
            1.19.1-1
    libnvidia-container1:
        ----------
        new:
            1.20.0-1
        old:
            1.19.1-1

This then updates the package.

Then, we see this:

salt-call --local state.single pkg.installed \
  name=test_nvidia_container_packages \
  pkgs='["libnvidia-container-tools", "libnvidia-container1:amd64"]' \
  update_holds=True
local:
----------
          ID: test_nvidia_container_packages
    Function: pkg.installed
      Result: True
     Comment: 1 targeted package was installed/updated.
              The following packages were already installed: libnvidia-container-tools
     Started: 14:40:04.574873
    Duration: 4158.501 ms
     Changes:   

Summary for local
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:   4.159 s

So this confirms that salt doesn't detect the installed package properly and calls apt for no reason.

Type of salt install

Official deb

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-24.04

salt --versions-report output
Salt Version:
          Salt: 3006.26
 
Python Version:
        Python: 3.10.20 (main, Jun 11 2026, 02:19:33) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.17.1
      cherrypy: 18.10.0
  cryptography: 47.0.0
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.44
        Jinja2: 3.1.6
       libgit2: 1.9.0
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.1.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: 1.18.0
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.14
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.5
 
System Versions:
          dist: ubuntu 24.04.2 noble
        locale: utf-8
       machine: x86_64
       release: 6.8.0-136-generic
        system: Linux
       version: Ubuntu 24.04.2 noble

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at Salt's pkg.install handling for the apt backend and reproduce the reported commands with libnvidia-container-tools and libnvidia-container1:amd64 on Ubuntu 24.04. Compare installed-package detection with and without split_arch, then verify that packages already installed at the requested state cause no unnecessary apt invocation.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python, ubuntu
Domain
devops, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.