Error bootlooping vttablet when enabling acls on vttablet with Vitess v20.0.1 and vitess-operator v2.13.1

I am running a test Vitess Cluster in EKS and have run into an issue when adding static authentication to the vtgate moving from --mysql_auth_server_impl=none to --mysql_auth_server_impl=static (+ the file and the reload interval), namely 1 of the 2 vttablets begins bootlooping.

This is the vtgate config https://gist.github.com/dan-frincu-udemy/2ab3797a01ce82438602789089da7ca4
Working vttablet config https://gist.github.com/dan-frincu-udemy/0767a9636030db953bb0e5685540f825
Failing vttablet config https://gist.github.com/dan-frincu-udemy/1a1736217ddcca1157e96063be89f1d4

Worth noting that
a) the config between vttablets is identical other than being deployed to different zones
b) if I remove the second vttablet and leave just the first, the same failure behaviour is now observed on the first, and when I add the 2nd tablet back, it moves the failure to the 2nd tablet.

The tablet that is working has the following in the describe output
The following is output from kubectl describe for

• working tablet https://gist.github.com/dan-frincu-udemy/9af91aa4ae43e3fd0758365687a2459a
• not working tablet https://gist.github.com/dan-frincu-udemy/45ad2379bdf2bf11650b7643d0f6e861

Because of the bootlooping of the tablet, we get 2 different kinds of logs, the 2nd one is potentially more relevant

• https://gist.github.com/dan-frincu-udemy/6f6dd985e81977c80878bb9c685eb729
• https://gist.github.com/dan-frincu-udemy/43ce75422ada916ec77b819259e0c39b

The mysql log https://gist.github.com/dan-frincu-udemy/629c75973f9bebe7036b1eb31b2ef5f6

Other observations:

• there are logs mentioning that /vt/socket/mysql.sock is missing on the vttablet that has errors, that is correct, only mysqlctl.sock is present in /vt/socket directory, but at the same time the mysqld server is running, makes a socket on 3306 which is reachable from outside the pod and the process on the pod is seen running

1 /vt/bin/mysqlctld --db_charset=utf8mb4 --db_dba_user=vt_dba --init_db_sql_file=/vt/secrets/db-init-script/init_db.sql --logtostderr=true --mysql_socket=/vt/socket/mysql.sock --socket_file=/vt/socket/mysqlctl.sock --tablet_uid=2169852485 --wait_time=2h0m0s
17 /usr/bin/sh /usr/bin/mysqld_safe --defaults-file=/vt/vtdataroot/vt_2169852485/my.cnf --basedir=/usr
539 /usr/sbin/mysqld --defaults-file=/vt/vtdataroot/vt_2169852485/my.cnf --basedir=/usr --datadir=/vt/vtdataroot/vt_2169852485/data --plugin-dir=/usr/lib64/mysql/plugin --log-error=/vt/config/stderr.symlink --pid-file=/vt/vtdataroot/vt_2169852485/mysql.pid --socket=/vt/socket/mysql.sock --port=3306

• the vitess user is not present on the mysqld container, so it shows UID 999 for the files in /vt/socket (this may not be a problem per se, the UID is correct)

Start by comparing the linked working and failing vttablet configs and kubectl describe outputs, then read both linked bootloop logs alongside the MySQL log. Trace the change from mysql_auth_server_impl=none to static and verify the socket and process observations; done means identifying the cause and preventing the affected vttablet from bootlooping with ACL authentication enabled.