Tooz's redis does not appear to respect username setting

Hello. I wanted to check to see if I'm doing something wrong, or if this comment should just be updated to not suggest including a username for a redis connection:

https://github.com/StackStorm/st2/blob/63f66b9595bf5b4e3974e7c5f327fa59f864ba3d/st2common/st2common/services/coordination.py#L206

But, when trying to connect to a Redis instance using a username, I get this error: "tooz.ToozError: WRONGPASS invalid username-password pair"

I set a debugger where the Tooz Redis driver creates the Redis client, and this is what I found:

(Pdb) locals()
{'cls': <class 'tooz.drivers.redis.RedisDriver'>, 'parsed_url': _ModifiedSplitResult(scheme='redis', netloc='stackstorm_appuser:[password removed]@redis-[hostname]:6379', path='', query='', fragment=''), 'options': {'lock_timeout': 60}, 'default_socket_timeout': 30, 'kwargs': {'host': 'redis-<>', 'port': 6379, 'password': '<>, 'socket_timeout': 30}, 'a': 'sentinel', 'pdb': <module 'pdb' from '/usr/lib/python3.8/pdb.py'>}
(Pdb) c
2023-01-06 23:48:28,185 ERROR [-] (PID=1575) ST2 API quit due to exception.
Traceback (most recent call last):
File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/drivers/redis.py", line 42, in _translate_failures
yield
File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/drivers/redis.py", line 452, in _start
self._server_info = self._client.info()
File "/opt/stackstorm/st2/lib/python3.8/site-packages/redis/client.py", line 1304, in info
return self.execute_command('INFO')
File "/opt/stackstorm/st2/lib/python3.8/site-packages/redis/client.py", line 898, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "/opt/stackstorm/st2/lib/python3.8/site-packages/redis/connection.py", line 1192, in get_connection
connection.connect()
File "/opt/stackstorm/st2/lib/python3.8/site-packages/redis/connection.py", line 567, in connect
self.on_connect()
File "/opt/stackstorm/st2/lib/python3.8/site-packages/redis/connection.py", line 643, in on_connect
auth_response = self.read_response()
File "/opt/stackstorm/st2/lib/python3.8/site-packages/redis/connection.py", line 756, in read_response
raise response
redis.exceptions.ResponseError: WRONGPASS invalid username-password pair

However, if I coerce the username into the kwargs (the Redis client does have the ability to accept and use the username), it works fine:

(Pdb) kwargs["username"]="stackstorm_appuser"
(Pdb) c
2023-01-06 23:47:17,499 INFO [-] Connected to amqp://guest:@rabbitmq:5672//
2023-01-06 23:47:17,612 INFO [-] Connected to amqp://guest:@rabbitmq:5672//

The Tooz redis driver doesn't seem to actually ever try to set a username (its documentation doesn't actually include username in the pattern at all): https://github.com/openstack/tooz/blob/master/tooz/drivers/redis.py#L162

Here is where the password is set, but there's no similar setting of the username: https://github.com/openstack/tooz/blob/master/tooz/drivers/redis.py#L424

Has anybody used a Redis connection with a non-default username and gotten it to work? Or, am I doing something horribly wrong here?

Start with st2common/st2common/services/coordination.py around line 206, then compare how tooz's Redis driver parses the URL near line 162 and builds connection arguments near line 424. Verify whether a non-default Redis username reaches the client; done means the supported behavior and the corresponding guidance are established for username-authenticated connections.