[BUG] salt-master service fails to start when using Vault SDB URI in config file
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
When using an SDB URI within a salt master config file to retrieve secret data from a Hashicorp Vault instance, salt-master service fails to start
Setup
- MongoDB for external pillar data is configured in /etc/salt/master.d/mongo_setup.conf
- MongoDB configuration requires the inclusion of the mongo user password.
- Password is replaced with a SDB URI to pull the password from a Hashicorp Vault instance, and salt-master service fails to start when restarted to apply the new configuration.
- SDB URI works successfully when used in pillar data, and when called directly using sdb.get runner
Please be as specific as possible and give set-up details.
Salt master is running as a CentOS7 VM
Steps to Reproduce the behavior
SDB URI configured in /etc/salt/master.d/mongo_setup.conf
mongo.host: 127.0.0.1
mongo.port: 27017
mongo.user: salt_lab_mongouser
mongo.username: salt_lab_mongouser
mongo.password: 'sdb://ssvault/salt/salt_lab_mongouser/password'
# ext_pillar setup
ext_pillar:
- mongo2: {collection: pillar_minion, id_field: minion_id, fields: [proxy]}
Vault configuration in /etc/salt/master.d/vault.conf
# Local Vault configuration
vault:
url: https://<REMOVED>
verify: '/opt/vault/tls/saltvault.crt'
auth:
method: token
token: <REMOVED>
Vault driver configured in /etc/salt/master:
##### Primary configuration settings #####
##########################################
# This configuration file is used to manage the behavior of the Salt Master.
# Values that are commented out but have an empty line after the comment are
# defaults that do not need to be set in the config. If there is no blank line
# after the comment then the value is presented as an example and is not the
# default.
ssvault:
driver: vault
...
<TRUNCATED>
...
Confirm SDB URI returns the expected result using sdb.get runner:
[root@salt-master-1 ~]# salt-run sdb.get 'sdb://ssvault/salt/salt_lab_mongouser/password'
$tr0ngP@$$w0rd
Restart salt-master service to apply config:
[root@salt-master-1 ~]# systemctl restart salt-master
Job for salt-master.service failed because the control process exited with error code. See "systemctl status salt-master.service" and "journalctl -xe" for details.
[root@salt-master-1 ~]# systemctl status salt-master -l
● salt-master.service - The Salt Master Server
Loaded: loaded (/usr/lib/systemd/system/salt-master.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2022-05-25 16:05:39 PDT; 12s ago
Docs: man:salt-master(1)
file:///usr/share/doc/salt/html/contents.html
https://docs.saltproject.io/en/latest/contents.html
Process: 15320 ExecStart=/usr/bin/salt-master (code=exited, status=64)
Main PID: 15320 (code=exited, status=64)
May 25 16:05:39 salt-master-1 salt-master[15320]: path_metadata = _get_secret_path_metadata(path)
May 25 16:05:39 salt-master-1 salt-master[15320]: File "/usr/lib/python3.6/site-packages/salt/utils/vault.py", line 508, in _get_secret_path_metadata
May 25 16:05:39 salt-master-1 salt-master[15320]: cache_content = _read_cache_file()
May 25 16:05:39 salt-master-1 salt-master[15320]: File "/usr/lib/python3.6/site-packages/salt/utils/vault.py", line 256, in _read_cache_file
May 25 16:05:39 salt-master-1 salt-master[15320]: return salt.utils.json.load(contents)
May 25 16:05:39 salt-master-1 salt-master[15320]: AttributeError: module 'salt.utils' has no attribute 'json'
May 25 16:05:39 salt-master-1 systemd[1]: salt-master.service: main process exited, code=exited, status=64/n/a
May 25 16:05:39 salt-master-1 systemd[1]: Failed to start The Salt Master Server.
May 25 16:05:39 salt-master-1 systemd[1]: Unit salt-master.service entered failed state.
May 25 16:05:39 salt-master-1 systemd[1]: salt-master.service failed.
Expected behavior
On restart of salt-master service, salt master should retrieve secret data from Vault instance and load it successfully as part of the external pillar configuration.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3004
Dependency Versions:
cffi: 1.15.0
cherrypy: unknown
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
python-gnupg: Not Installed
PyYAML: 6.0
PyZMQ: 17.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1160.31.1.el7.x86_64
system: Linux
version: CentOS Linux 7 Core
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read salt/utils/vault.py around _get_secret_path_metadata and _read_cache_file, then reproduce the failure with the shown SDB URI and salt-master restart. Confirm that the master starts successfully and loads the MongoDB password from Vault, while salt-run sdb.get continues to return the expected secret.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb, python
- Domain
- database, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100