[BUG] cannot use mysql for minion cache + job cache due to conflicting password config opts
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
mysql minion cache passes all unexpected 'mysql.*' options to the connection.
See https://github.com/saltstack/salt/blob/c2779df50c64ef7b16dea4aeaa08eb5ebad3d12b/salt/cache/mysql_cache.py#L248-L252
Meanwhile, the mysql returner uses mysql.pass to read the password, but pass is not a valid option for the python lib python-mysqldb: https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html - only "password" or "passwd" are valid.
A simple change might be to set tthe mysql returner to use mysql.password, so that the same password config can be read by both returner and cache?
Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Please be as specific as possible and give set-up details.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Setup the mysql returner in master config:
master_job_cache: mysql
mysql.host: '127.0.0.1'
mysql.user: 'read-write'
mysql.pass: 'somepassword'
mysql.db: 'salt'
mysql.port: 3306
setup the mysql cache in master config:
cache: mysql
mysql.database: 'salt'
mysql.table_name: 'minion_cache'
mysql.password: 'somepassword'
Steps to Reproduce the behavior
Causes an error when cache connection is created when master starts and minions attempt to connect, create jobs:
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: [DEBUG ] mysql_cache: creating db connection
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: [ERROR ] An un-handled exception from the multiprocessing process 'Maintenance' was caught:
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: Traceback (most recent call last):
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/cache/mysql_cache.py", line 117, in run_query
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: cur = conn.cursor()
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: AttributeError: 'NoneType' object has no attribute 'cursor'
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: During handling of the above exception, another exception occurred:
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: Traceback (most recent call last):
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/utils/process.py", line 993, in wrapped_run_func
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: return run_func()
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/master.py", line 265, in run
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: self.handle_presence(old_present)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/master.py", line 357, in handle_presence
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: present = self.ckminions.connected_ids()
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/utils/minions.py", line 634, in connected_ids
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: search = self.cache.list("minions")
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/cache/__init__.py", line 233, in list
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: return self.modules[fun](bank, **self._kwargs)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: return self.loader.run(run_func, *args, **kwargs)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1228, in run
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1243, in _run_as
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: return _func_or_method(*args, **kwargs)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/cache/mysql_cache.py", line 309, in ls
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: _init_client()
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/cache/mysql_cache.py", line 252, in _init_client
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: _create_table()
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/cache/mysql_cache.py", line 161, in _create_table
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: cur, _ = run_query(
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/salt/cache/mysql_cache.py", line 136, in run_query
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: __context__["mysql_client"] = MySQLdb.connect(**__context__["mysql_kwargs"])
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/MySQLdb/__init__.py", line 84, in Connect
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: return Connection(*args, **kwargs)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 179, in __init__
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: super(Connection, self).__init__(*args, **kwargs2)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: TypeError: 'pass' is an invalid keyword argument for connect()
Expected behavior
Mysql connection should be created for job cache/returner AND minion cache
Screenshots
n/a
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)(root) ~ # salt -V
Salt Version:
Salt: 3005.3
Dependency Versions:
cffi: 1.15.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.0.3
libgit2: 1.1.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.3
msgpack-pure: Not Installed
mysql-python: 1.4.6
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.11.0
pygit2: 1.6.1
Python: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 22.3.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 22.04 Jammy Jellyfish
locale: utf-8
machine: x86_64
release: 5.15.112netflix-g7505e3b8e9c5
system: Linux
version: Ubuntu 22.04 Jammy Jellyfish
Additional context
n/a
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with salt/cache/mysql_cache.py around lines 248-252 and the mysql returner’s password configuration handling. Trace how mysql.* options become MySQLdb.connect() arguments, then determine a consistent password option for both cache and returner. Done means the supplied configuration creates both MySQL connections without passing the invalid 'pass' keyword.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100