crossplane-contrib / crossplane-contrib/provider-sql

MySQL: pending ALTER USER/GRANT statements pile up and make the server unresponsive

Open
#422 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
154
Forks
119
Avg merge
6d 17h
Merged PRs (30d)
8

Description

## What happened

Under a backlog of pending `ALTER USER` (and other write) statements issued by provider-sql, a MySQL/Percona server can become unresponsive — `SHOW PROCESSLIST` fills with account-management statements stuck in `Waiting for metadata lock`, and the connection count climbs until `max_connections` is exhausted.

## Root cause

1. **The driver never issues `KILL` on context cancel.** `github.com/go-sql-driver/mysql` cancels a `context` by closing the TCP socket, **not** by sending `KILL QUERY`. This is intentional and present in every release including the latest `v1.10.0`, so it cannot be fixed by upgrading. The statement keeps running/waiting server-side.
2. **`lock_wait_timeout` default is 1 year.** A statement blocked on a metadata/ACL lock (e.g. behind a backup such as Percona XtraBackup / `FLUSH TABLES WITH READ LOCK`, or a long transaction) stays pending on the server effectively forever.
3. **The reconcile deadline (60s, crossplane-runtime default) only closes the socket** — see (1) — so the provider gives up but the server thread keeps waiting.
4. **Retries re-issue the statement.** Each reconcile opens a fresh connection (the MySQL client calls `sql.Open` per query) and issues the statement again, which also blocks.
5. **Account-management statements serialize on a single global ACL lock**, so one stuck `GRANT`/`ALTER USER` stalls *every* account-management statement server-wide.

Net effect: blocked statements and their connections accumulate until the server is unresponsive. Affects all MySQL controllers (User, Grant, Database; cluster and namespaced) since they share the client. Not specific to password rotation — any `ALTER USER`/`GRANT`/DDL can trigger it.

## Proposed fix

Append `lock_wait_timeout` (and a dial `timeout`) to the MySQL DSN so a blocked statement fails fast server-side and releases instead of piling up. PR incoming.

## Not covered

- Non-lock hangs (dead network, server wedged mid-execution) still only get socket-closed-without-`KILL`; an application-level `KILL QUERY` pattern would be needed for that class.
- The per-query `sql.Open` / unbounded connection design is a separate concern (bounded shared pool).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the shared MySQL client and the per-query sql.Open path used by the User, Grant, and Database controllers, then inspect how the DSN is constructed. Done means blocked account-management statements fail fast server-side with the proposed lock and dial timeouts instead of accumulating after reconcile cancellation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, mysql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.