crossplane-contrib / crossplane-contrib/provider-sql
MySQL: pending ALTER USER/GRANT statements pile up and make the server unresponsive
- Dominant language
- Go
- Stars
- 154
- Forks
- 119
- Avg merge
- 6d 17h
- Merged PRs (30d)
- 8
Description
## What happened
Under a backlog of pending `ALTER USER` (and other write) statements issued by provider-sql, a MySQL/Percona server can become unresponsive — `SHOW PROCESSLIST` fills with account-management statements stuck in `Waiting for metadata lock`, and the connection count climbs until `max_connections` is exhausted.
## Root cause
1. **The driver never issues `KILL` on context cancel.** `github.com/go-sql-driver/mysql` cancels a `context` by closing the TCP socket, **not** by sending `KILL QUERY`. This is intentional and present in every release including the latest `v1.10.0`, so it cannot be fixed by upgrading. The statement keeps running/waiting server-side.
2. **`lock_wait_timeout` default is 1 year.** A statement blocked on a metadata/ACL lock (e.g. behind a backup such as Percona XtraBackup / `FLUSH TABLES WITH READ LOCK`, or a long transaction) stays pending on the server effectively forever.
3. **The reconcile deadline (60s, crossplane-runtime default) only closes the socket** — see (1) — so the provider gives up but the server thread keeps waiting.
4. **Retries re-issue the statement.** Each reconcile opens a fresh connection (the MySQL client calls `sql.Open` per query) and issues the statement again, which also blocks.
5. **Account-management statements serialize on a single global ACL lock**, so one stuck `GRANT`/`ALTER USER` stalls *every* account-management statement server-wide.
Net effect: blocked statements and their connections accumulate until the server is unresponsive. Affects all MySQL controllers (User, Grant, Database; cluster and namespaced) since they share the client. Not specific to password rotation — any `ALTER USER`/`GRANT`/DDL can trigger it.
## Proposed fix
Append `lock_wait_timeout` (and a dial `timeout`) to the MySQL DSN so a blocked statement fails fast server-side and releases instead of piling up. PR incoming.
## Not covered
- Non-lock hangs (dead network, server wedged mid-execution) still only get socket-closed-without-`KILL`; an application-level `KILL QUERY` pattern would be needed for that class.
- The per-query `sql.Open` / unbounded connection design is a separate concern (bounded shared pool).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the shared MySQL client and the per-query sql.Open path used by the User, Grant, and Database controllers, then inspect how the DSN is constructed. Done means blocked account-management statements fail fast server-side with the proposed lock and dial timeouts instead of accumulating after reconcile cancellation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, mysql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100