GoogleCloudPlatform / GoogleCloudPlatform/cloud-sql-nodejs-connector

Connector.close() + pool.end() leaks TLS sockets for in-flight queries

Open
#562 5 comments 2 reactions 0 assignees View on GitHub
priority: p1 type: bug
Dominant language
TypeScript
Stars
97
Forks
16
Avg merge
4d 16h
Merged PRs (30d)
4

Description

## Bug Description

`Connector.close()` followed by `pool.end()` leaves TLS sockets alive for any mysql2 pool connection that was mid-query when shutdown began. The leak is 1:1 with the number of in-flight queries: with 5 concurrent `SELECT SLEEP(2)` queries running, `whyIsNodeRunning()` 500 ms after shutdown reports 5 `TLSWRAP` handles still pinning the event loop. Their stacks root at:

```
@google-cloud/cloud-sql-connector/dist/mjs/socket.js:107 — tls.connect(socketOpts)
@google-cloud/cloud-sql-connector/dist/mjs/connector.js:148 — getSocket(...)
mysql2/lib/base/connection.js:64 — this.stream = opts.config.stream(opts)
```

Once the socket count grows past a few (e.g. a serving process handling concurrent traffic at the moment `SIGTERM` arrives), the process can fail to exit on graceful shutdown, relying on a forced kill.

Reproduces on both `@google-cloud/cloud-sql-connector@1.9.2` and `@google-cloud/cloud-sql-connector@1.10.0` (latest as of filing).

Suspected cause: `CloudSQLInstance.close()` destroys sockets in `this.sockets`, but pool connections that are currently checked out (mid-query) don't appear to be tracked there. Additionally, `Connector.close()` is synchronous but internally fire-and-forget (`instance.promise.then(inst => inst.close())`), so caller code that immediately follows `close()` with `pool.end()` races the microtask that actually destroys instance sockets.

## Example code (or command)

```js
import mysql from 'mysql2/promise';
import { AuthTypes, Connector } from '@google-cloud/cloud-sql-connector';
import whyIsNodeRunning from 'why-is-node-running';

const INSTANCE_CONNECTION_NAME = 'TODO: your-project:region:instance';
const DATABASE = 'TODO: your-database';
const USER = 'TODO: your-iam-user';

async function main() {
const connector = new Connector();
const clientOpts = await connector.getOptions({
instanceConnectionName: INSTANCE_CONNECTION_NAME,
authType: AuthTypes.IAM,
});
const pool = mysql.createPool({
...clientOpts,
user: USER,
database: DATABASE,
});

// Fire concurrent in-flight queries without awaiting — simulates request
// handlers still running when shutdown begins.
const queries = Array.from({ length: 5 }, (_, i) =>
pool.query('SELECT SLEEP(2), ?', [i]).catch(() => {})
);

// Let the queries actually open TLS sockets.
await new Promise((r) => setTimeout(r, 100));

await pool.end();
connector.close();

// Let libuv settle (collect deactivated timers, flush pending writes).
await new Promise((r) => setTimeout(r, 500));

whyIsNodeRunning();

await Promise.allSettled(queries);
}

main();
```

## Stacktrace

Output of `whyIsNodeRunning()` 500 ms after `pool.end()` + `connector.close()`:

```
% node temp.direct.mjs

...

# TLSWRAP
node_modules/.pnpm/@google-cloud+cloud-sql-connector@1.10.0/node_modules/@google-cloud/cloud-sql-connector/dist/mjs/socket.js:107 - const tlsSocket = tls.connect(socketOpts);
node_modules/.pnpm/@google-cloud+cloud-sql-connector@1.10.0/node_modules/@google-cloud/cloud-sql-connector/dist/mjs/connector.js:148 - const tlsSocket = getSocket({
node_modules/.pnpm/mysql2@3.16.3/node_modules/mysql2/lib/base/connection.js:64 - this.stream = opts.config.stream(opts);
node_modules/.pnpm/mysql2@3.16.3/node_modules/mysql2/lib/base/pool_connection.js:7 - super(options);
node_modules/.pnpm/mysql2@3.16.3/node_modules/mysql2/lib/pool_connection.js:5 - class PoolConnection extends BasePoolConnection {

...

# FILEHANDLE
(unknown stack trace)

# TTYWRAP
temp.direct.mjs:40 - console.log('shutting down while queries are in flight...');

# Timeout
temp.direct.mjs:46 - await new Promise((r) => setTimeout(r, 500));
temp.direct.mjs:46 - await new Promise((r) => setTimeout(r, 500));
```

## How to reproduce

1. `npm init -y && npm pkg set type=module`
2. `npm i @google-cloud/cloud-sql-connector@1.10.0 mysql2@3.16.3 why-is-node-running`
3. Save the example code above as `repro.mjs`, filling in `INSTANCE_CONNECTION_NAME`, `DATABASE`, `USER`.
4. Ensure `gcloud auth application-default login` has been run and the IAM user has access to the instance.
5. `node repro.mjs`
6. Observe `whyIsNodeRunning()` reports N `TLSWRAP` handles rooted at `@google-cloud/cloud-sql-connector/dist/mjs/socket.js` — one per in-flight query that hadn't completed when `pool.end()` was called.

#### Environment details

- OS: MacOS 15.7.4 ARM (M2 Pro)
- Node.js version: v20.20.0 and v24.13.0
- pnpm version: 10.22.0
- `@google-cloud/cloud-sql-connector` version: 1.10.0
- `mysql2` version: 3.16.3

#### Steps to reproduce

1. Create a `Connector` and a `mysql2` pool per the package README.
2. Issue several concurrent queries without awaiting them.
3. Before the queries resolve, call `await pool.end()` and `connector.close()`.
4. Wait long enough (≥ 500 ms) for libuv to collect any handles that were legitimately cleared.
5. Call `whyIsNodeRunning()` (or otherwise inspect the event loop).
6. Expected: no `TLSWRAP` handles from the connector. Actual: one leaked `TLSWRAP` per in-flight query at shutdown time.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.