saltstack / saltstack/salt

[BUG] salt-master service fails to start when using Vault SDB URI in config file

Open
#62,113 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
When using an SDB URI within a salt master config file to retrieve secret data from a Hashicorp Vault instance, salt-master service fails to start

Setup

  1. MongoDB for external pillar data is configured in /etc/salt/master.d/mongo_setup.conf
  2. MongoDB configuration requires the inclusion of the mongo user password.
  3. Password is replaced with a SDB URI to pull the password from a Hashicorp Vault instance, and salt-master service fails to start when restarted to apply the new configuration.
  4. SDB URI works successfully when used in pillar data, and when called directly using sdb.get runner

Please be as specific as possible and give set-up details.

Salt master is running as a CentOS7 VM

Steps to Reproduce the behavior

SDB URI configured in /etc/salt/master.d/mongo_setup.conf

mongo.host: 127.0.0.1
mongo.port: 27017
mongo.user: salt_lab_mongouser
mongo.username: salt_lab_mongouser
mongo.password: 'sdb://ssvault/salt/salt_lab_mongouser/password'

# ext_pillar setup
ext_pillar:
   - mongo2: {collection: pillar_minion, id_field: minion_id,  fields: [proxy]}

Vault configuration in /etc/salt/master.d/vault.conf

# Local Vault configuration
vault:
  url: https://<REMOVED>
  verify: '/opt/vault/tls/saltvault.crt'
  auth:
    method: token
    token: <REMOVED>

Vault driver configured in /etc/salt/master:

##### Primary configuration settings #####
##########################################
# This configuration file is used to manage the behavior of the Salt Master.
# Values that are commented out but have an empty line after the comment are
# defaults that do not need to be set in the config. If there is no blank line
# after the comment then the value is presented as an example and is not the
# default.

ssvault:
  driver: vault
... 
<TRUNCATED>
...

Confirm SDB URI returns the expected result using sdb.get runner:

[root@salt-master-1 ~]# salt-run sdb.get 'sdb://ssvault/salt/salt_lab_mongouser/password'

$tr0ngP@$$w0rd

Restart salt-master service to apply config:

[root@salt-master-1 ~]# systemctl restart salt-master
Job for salt-master.service failed because the control process exited with error code. See "systemctl status salt-master.service" and "journalctl -xe" for details.


[root@salt-master-1 ~]# systemctl status salt-master -l
● salt-master.service - The Salt Master Server
   Loaded: loaded (/usr/lib/systemd/system/salt-master.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2022-05-25 16:05:39 PDT; 12s ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltproject.io/en/latest/contents.html
  Process: 15320 ExecStart=/usr/bin/salt-master (code=exited, status=64)
 Main PID: 15320 (code=exited, status=64)

May 25 16:05:39 salt-master-1 salt-master[15320]: path_metadata = _get_secret_path_metadata(path)
May 25 16:05:39 salt-master-1 salt-master[15320]: File "/usr/lib/python3.6/site-packages/salt/utils/vault.py", line 508, in _get_secret_path_metadata
May 25 16:05:39 salt-master-1 salt-master[15320]: cache_content = _read_cache_file()
May 25 16:05:39 salt-master-1 salt-master[15320]: File "/usr/lib/python3.6/site-packages/salt/utils/vault.py", line 256, in _read_cache_file
May 25 16:05:39 salt-master-1 salt-master[15320]: return salt.utils.json.load(contents)
May 25 16:05:39 salt-master-1 salt-master[15320]: AttributeError: module 'salt.utils' has no attribute 'json'
May 25 16:05:39 salt-master-1 systemd[1]: salt-master.service: main process exited, code=exited, status=64/n/a
May 25 16:05:39 salt-master-1 systemd[1]: Failed to start The Salt Master Server.
May 25 16:05:39 salt-master-1 systemd[1]: Unit salt-master.service entered failed state.
May 25 16:05:39 salt-master-1 systemd[1]: salt-master.service failed.

Expected behavior
On restart of salt-master service, salt master should retrieve secret data from Vault instance and load it successfully as part of the external pillar configuration.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3004

Dependency Versions:
          cffi: 1.15.0
      cherrypy: unknown
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 6.0
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1160.31.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read salt/utils/vault.py around _get_secret_path_metadata and _read_cache_file, then reproduce the failure with the shown SDB URI and salt-master restart. Confirm that the master starts successfully and loads the MongoDB password from Vault, while salt-run sdb.get continues to return the expected secret.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
database, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.