saltstack / saltstack/salt

[BUG] cannot use mysql for minion cache + job cache due to conflicting password config opts

Open
#65,321 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
mysql minion cache passes all unexpected 'mysql.*' options to the connection.
See https://github.com/saltstack/salt/blob/c2779df50c64ef7b16dea4aeaa08eb5ebad3d12b/salt/cache/mysql_cache.py#L248-L252

Meanwhile, the mysql returner uses mysql.pass to read the password, but pass is not a valid option for the python lib python-mysqldb: https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html - only "password" or "passwd" are valid.

A simple change might be to set tthe mysql returner to use mysql.password, so that the same password config can be read by both returner and cache?

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Setup the mysql returner in master config:

master_job_cache: mysql
mysql.host: '127.0.0.1'
mysql.user: 'read-write'
mysql.pass: 'somepassword'
mysql.db: 'salt'
mysql.port: 3306

setup the mysql cache in master config:

cache: mysql
mysql.database: 'salt'
mysql.table_name: 'minion_cache'
mysql.password: 'somepassword'

Steps to Reproduce the behavior

Causes an error when cache connection is created when master starts and minions attempt to connect, create jobs:

Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: [DEBUG   ] mysql_cache: creating db connection
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: [ERROR   ] An un-handled exception from the multiprocessing process 'Maintenance' was caught:
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: Traceback (most recent call last):
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/cache/mysql_cache.py", line 117, in run_query
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     cur = conn.cursor()
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: AttributeError: 'NoneType' object has no attribute 'cursor'
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: During handling of the above exception, another exception occurred:
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: Traceback (most recent call last):
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/utils/process.py", line 993, in wrapped_run_func
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     return run_func()
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/master.py", line 265, in run
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     self.handle_presence(old_present)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/master.py", line 357, in handle_presence
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     present = self.ckminions.connected_ids()
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/utils/minions.py", line 634, in connected_ids
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     search = self.cache.list("minions")
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/cache/__init__.py", line 233, in list
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     return self.modules[fun](bank, **self._kwargs)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     return self.loader.run(run_func, *args, **kwargs)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1228, in run
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1243, in _run_as
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     return _func_or_method(*args, **kwargs)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/cache/mysql_cache.py", line 309, in ls
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     _init_client()
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/cache/mysql_cache.py", line 252, in _init_client
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     _create_table()
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/cache/mysql_cache.py", line 161, in _create_table
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     cur, _ = run_query(
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/salt/cache/mysql_cache.py", line 136, in run_query
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     __context__["mysql_client"] = MySQLdb.connect(**__context__["mysql_kwargs"])
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/MySQLdb/__init__.py", line 84, in Connect
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     return Connection(*args, **kwargs)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:   File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 179, in __init__
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]:     super(Connection, self).__init__(*args, **kwargs2)
Sep 29 20:23:54 d0ff1c24-e6b7-4216-9cb8-193f0f132e62 salt-master[65863]: TypeError: 'pass' is an invalid keyword argument for connect()

Expected behavior
Mysql connection should be created for job cache/returner AND minion cache

Screenshots
n/a

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
(root) ~ # salt -V
Salt Version:
          Salt: 3005.3
 
Dependency Versions:
          cffi: 1.15.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.3
       libgit2: 1.1.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.3
  msgpack-pure: Not Installed
  mysql-python: 1.4.6
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.11.0
        pygit2: 1.6.1
        Python: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 22.3.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 22.04 Jammy Jellyfish
        locale: utf-8
       machine: x86_64
       release: 5.15.112netflix-g7505e3b8e9c5
        system: Linux
       version: Ubuntu 22.04 Jammy Jellyfish

Additional context
n/a

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with salt/cache/mysql_cache.py around lines 248-252 and the mysql returner’s password configuration handling. Trace how mysql.* options become MySQLdb.connect() arguments, then determine a consistent password option for both cache and returner. Done means the supplied configuration creates both MySQL connections without passing the invalid 'pass' keyword.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.