[BUG] 3006.9 MySQL module failure

Description
Trying to use mysql state/execution modules, which now needs salt-pip to function in/with onedir.
I see #65980, which is marked as closed and resolved for 3006.9, but appears to be the closest issue to what I'm seeing. I tested all of the listed pip packages in the documentation.

Setup
I am using multipass to spin up quick VMs for testing across multiple distros. Below oneliner spins up an instance of 20.04, 22.04, and 24.04.
for dist in focal jammy noble ; do multipass launch -n pymy-$dist -d 10G $dist ; done

From there, I am running a very simple bootstrap shell script to stand up a mysql instance, as well as a salt-master and salt-minion.

#!/bin/bash
#get system uptodate
sudo apt-get update && sudo apt-get -y dist-upgrade

#setup salt repo
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://packages.broadcom.com/artifactory/api/security/keypair/SaltProjectKey/public | sudo tee /etc/apt/keyrings/salt-archive-keyring.pgp
curl -fsSL https://github.com/saltstack/salt-install-guide/releases/latest/download/salt.sources | sudo tee /etc/apt/sources.list.d/salt.sources
echo 'Package: salt-*
Pin: version 3006.*
Pin-Priority: 1001' | sudo tee /etc/apt/preferences.d/salt-pin-1001

#add 1GB swapfile - need more than 1G memory for mysql and salt-master
sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

#install salt
sudo apt-get update && sudo apt-get -y install salt-master salt-minion mysql-server
echo 'interface: 0.0.0.0' | sudo tee /etc/salt/master.d/master.conf
echo "master: $HOSTNAME" | sudo tee /etc/salt/minion.d/minion.conf
sudo systemctl restart salt-master.service salt-minion.service
sleep 30
sudo salt-key -A -y
sleep 15
sudo salt '*' test.ping

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
After the bootstrap script is run, I have a second shell script that runs through the options of:

• do nothing (No python mysql client installed)
• mysqlclient (salt-pip failure; No python mysql client installed)
• saltext-mysql (salt-pip success; No python mysql client installed)
• pymysql (salt-pip success; MySQL Error 1698: Access denied for user 'root'@'localhost' ; local: False)
• horrible workaround with devel packages ang mysqlclient (salt-pip success; local: true)

#!/bin/bash

echo "testing without anything"
sudo salt-call mysql.db_create 'testname'

echo "testing mysqlclient"
sudo salt-pip install mysqlclient
sudo salt-call mysql.db_create 'mysqlclient'

echo "testing saltext-mysql"
sudo salt-pip install saltext-mysql
sudo salt-call mysql.db_create 'saltext-mysql'

echo "testing pymysql"
sudo salt-pip install pymysql
sudo salt-call mysql.db_create 'pymysql'

echo "testing horrible workaround"
sudo apt-get -y install pkg-config python3-dev default-libmysqlclient-dev build-essential
sudo salt-pip install mysqlclient
sudo salt-call mysql.db_create 'mysqlclient-v2'

testing without anything
/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/psutil_compat.py:16: DeprecationWarning: Please stop importing 'salt.utils.psutil_compat' and instead import 'psutil' directly as there's no longer a need for a compatability layer. The 'salt.utils.psutil_compat' will go away on Salt 3008.0 (Argon).
  salt.utils.versions.warn_until(
'mysql' __virtual__ returned False: No python mysql client installed.
testing mysqlclient
Collecting mysqlclient
  Downloading mysqlclient-2.2.6.tar.gz (91 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 91.1/91.1 kB 3.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [30 lines of output]
      /bin/sh: 1: pkg-config: not found
      /bin/sh: 1: pkg-config: not found
      /bin/sh: 1: pkg-config: not found
      /bin/sh: 1: pkg-config: not found
      Trying pkg-config --exists mysqlclient
      Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127.
      Trying pkg-config --exists mariadb
      Command 'pkg-config --exists mariadb' returned non-zero exit status 127.
      Trying pkg-config --exists libmariadb
      Command 'pkg-config --exists libmariadb' returned non-zero exit status 127.
      Trying pkg-config --exists perconaserverclient
      Command 'pkg-config --exists perconaserverclient' returned non-zero exit status 127.
      Traceback (most recent call last):
        File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-2yscf30e/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
        File "/tmp/pip-build-env-2yscf30e/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-2yscf30e/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in run_setup
          exec(code, locals())
        File "<string>", line 155, in <module>
        File "<string>", line 49, in get_config_posix
        File "<string>", line 28, in find_package_name
      Exception: Can not find valid pkg-config name.
      Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[notice] A new release of pip is available: 23.3.2 -> 24.3.1
[notice] To update, run: /opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip
/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/psutil_compat.py:16: DeprecationWarning: Please stop importing 'salt.utils.psutil_compat' and instead import 'psutil' directly as there's no longer a need for a compatability layer. The 'salt.utils.psutil_compat' will go away on Salt 3008.0 (Argon).
  salt.utils.versions.warn_until(
'mysql' __virtual__ returned False: No python mysql client installed.
testing saltext-mysql
Collecting saltext-mysql
  Downloading saltext.mysql-1.0.0-py2.py3-none-any.whl.metadata (5.5 kB)
Requirement already satisfied: salt>=3006 in /opt/saltstack/salt/lib/python3.10/site-packages (from saltext-mysql) (3006.9)
Collecting sqlparse (from saltext-mysql)
  Downloading sqlparse-0.5.2-py3-none-any.whl.metadata (3.9 kB)
[SNIP]
Downloading saltext.mysql-1.0.0-py2.py3-none-any.whl (49 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.1/49.1 kB 2.0 MB/s eta 0:00:00
Downloading sqlparse-0.5.2-py3-none-any.whl (44 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.4/44.4 kB 4.5 MB/s eta 0:00:00
Installing collected packages: sqlparse, saltext-mysql
Successfully installed saltext-mysql sqlparse
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.3.2 -> 24.3.1
[notice] To update, run: /opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip
/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/psutil_compat.py:16: DeprecationWarning: Please stop importing 'salt.utils.psutil_compat' and instead import 'psutil' directly as there's no longer a need for a compatability layer. The 'salt.utils.psutil_compat' will go away on Salt 3008.0 (Argon).
  salt.utils.versions.warn_until(
'mysql' __virtual__ returned False: No python mysql client installed.
testing pymysql
Collecting pymysql
  Downloading PyMySQL-1.1.1-py3-none-any.whl.metadata (4.4 kB)
Downloading PyMySQL-1.1.1-py3-none-any.whl (44 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.0/45.0 kB 2.2 MB/s eta 0:00:00
Installing collected packages: pymysql
Successfully installed pymysql
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.3.2 -> 24.3.1
[notice] To update, run: /opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip
[ERROR   ] MySQL Error 1698: Access denied for user 'root'@'localhost'
[ERROR   ] MySQL Error 1698: Access denied for user 'root'@'localhost'
local:
    False
testing horrible workaround
[SNIP]
Collecting mysqlclient
  Using cached mysqlclient-2.2.6.tar.gz (91 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: mysqlclient
  Building wheel for mysqlclient (pyproject.toml) ... done
  Created wheel for mysqlclient: filename=mysqlclient-2.2.6-cp310-cp310-linux_x86_64.whl size=129144 sha256=202dfdc97281d044bb6da4f069419071b726a0ee26faede3972d504611eb4432
  Stored in directory: /root/.cache/pip/wheels/c0/95/75/9ec8105309b2316e3e5eb4d3f661fa392f1bef34597232cef9
Successfully built mysqlclient
Installing collected packages: mysqlclient
  WARNING: In `rpath_only mode` but /lib/x86_64-linux-gnu/libmysqlclient.so.21 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib/x86_64-linux-gnu/libssl.so.1.1 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib/x86_64-linux-gnu/libcrypto.so.1.1 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib/x86_64-linux-gnu/libstdc++.so.6 is not in /opt/saltstack/salt
  Do not adjust rpath of /opt/saltstack/salt/extras-3.10/MySQLdb/_mysql.cpython-310-x86_64-linux-gnu.so
Successfully installed mysqlclient
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.3.2 -> 24.3.1
[notice] To update, run: /opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip
local:
    True

Expected behavior
Expected behavior is salt-pip to "just work" since the whole point of onedir is that it is wholly decoupled from the underlying OS. I feel like having to install a bunch of devel packages is actually even worse than needing to manage system pip packages for the local python3 environment.

Versions Report

$ sudo salt-run -V
Salt Version:
          Salt: 3006.9

Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

Salt Extensions:
 saltext.mysql: 1.0.0

System Versions:
          dist: ubuntu 20.04.6 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-200-generic
        system: Linux
       version: Ubuntu 20.04.6 focal

Additional context

$ sudo salt-pip freeze
certifi==2024.7.4
cffi==1.14.6
charset-normalizer==3.2.0
cheroot==8.5.2
CherryPy==18.6.1
contextvars==2.4
croniter==2.0.5
cryptography==42.0.5
distro==1.5.0
idna==3.7
immutables==0.15
importlib-metadata==6.0.0
jaraco.classes==3.2.1
jaraco.collections==3.4.0
jaraco.functools==2.0
jaraco.text==3.5.1
Jinja2==3.1.4
jmespath==1.0.1
looseversion==1.0.2
MarkupSafe==2.1.2
more-itertools==5.0.0
msgpack==1.0.2
mysqlclient==2.2.6
packaging==22.0
portend==2.4
psutil==5.8.0
pycparser==2.21
pycryptodomex==3.19.1
PyMySQL==1.1.1
pyOpenSSL==24.0.0
python-dateutil==2.8.1
python-gnupg==0.4.8
pytz==2022.1
PyYAML==6.0.1
pyzmq==23.2.0
relenv @ file:///opt/actions-runner/_work/relenv/relenv
requests==2.32.3
rpm-vercmp==0.1.2
salt @ file:///opt/actions-runner/_work/salt/salt/salt-3006.9.tar.gz#sha256=7703c73a71c67327d48b2893ce9cf031ee52e2816df0daf78d6fe18a99b353e3
saltext.mysql==1.0.0
setproctitle==1.3.2
six==1.16.0
sqlparse==0.5.2
tempora==4.1.1
timelib==0.2.5
urllib3==1.26.18
zc.lockfile==1.4
zipp==3.6.0

Start with the salt.modules.mysql entry point and reproduce the listed salt-pip and mysql.db_create commands in the onedir setup. Compare the documented Python client requirements with the observed failures; done means the MySQL state and execution modules work in onedir without the described development-package workaround.