GoogleCloudPlatform / GoogleCloudPlatform/cloud-sql-nodejs-connector

Connector.close() + pool.end() leaks TLS sockets for in-flight queries

Open
#562 5 comments 2 reactions 0 assignees View on GitHub
priority: p1 type: bug
Dominant language
TypeScript
Stars
97
Forks
16
Avg merge
4d 16h
Merged PRs (30d)
4

Description

## Bug Description

`Connector.close()` followed by `pool.end()` leaves TLS sockets alive for any mysql2 pool connection that was mid-query when shutdown began. The leak is 1:1 with the number of in-flight queries: with 5 concurrent `SELECT SLEEP(2)` queries running, `whyIsNodeRunning()` 500 ms after shutdown reports 5 `TLSWRAP` handles still pinning the event loop. Their stacks root at:

```
@google-cloud/cloud-sql-connector/dist/mjs/socket.js:107 — tls.connect(socketOpts)
@google-cloud/cloud-sql-connector/dist/mjs/connector.js:148 — getSocket(...)
mysql2/lib/base/connection.js:64 — this.stream = opts.config.stream(opts)
```

Once the socket count grows past a few (e.g. a serving process handling concurrent traffic at the moment `SIGTERM` arrives), the process can fail to exit on graceful shutdown, relying on a forced kill.

Reproduces on both `@google-cloud/cloud-sql-connector@1.9.2` and `@google-cloud/cloud-sql-connector@1.10.0` (latest as of filing).

Suspected cause: `CloudSQLInstance.close()` destroys sockets in `this.sockets`, but pool connections that are currently checked out (mid-query) don't appear to be tracked there. Additionally, `Connector.close()` is synchronous but internally fire-and-forget (`instance.promise.then(inst => inst.close())`), so caller code that immediately follows `close()` with `pool.end()` races the microtask that actually destroys instance sockets.

## Example code (or command)

```js
import mysql from 'mysql2/promise';
import { AuthTypes, Connector } from '@google-cloud/cloud-sql-connector';
import whyIsNodeRunning from 'why-is-node-running';

const INSTANCE_CONNECTION_NAME = 'TODO: your-project:region:instance';
const DATABASE = 'TODO: your-database';
const USER = 'TODO: your-iam-user';

async function main() {
const connector = new Connector();
const clientOpts = await connector.getOptions({
instanceConnectionName: INSTANCE_CONNECTION_NAME,
authType: AuthTypes.IAM,
});
const pool = mysql.createPool({
...clientOpts,
user: USER,
database: DATABASE,
});

// Fire concurrent in-flight queries without awaiting — simulates request
// handlers still running when shutdown begins.
const queries = Array.from({ length: 5 }, (_, i) =>
pool.query('SELECT SLEEP(2), ?', [i]).catch(() => {})
);

// Let the queries actually open TLS sockets.
await new Promise((r) => setTimeout(r, 100));

await pool.end();
connector.close();

// Let libuv settle (collect deactivated timers, flush pending writes).
await new Promise((r) => setTimeout(r, 500));

whyIsNodeRunning();

await Promise.allSettled(queries);
}

main();
```

## Stacktrace

Output of `whyIsNodeRunning()` 500 ms after `pool.end()` + `connector.close()`:

```
% node temp.direct.mjs

...

# TLSWRAP
node_modules/.pnpm/@google-cloud+cloud-sql-connector@1.10.0/node_modules/@google-cloud/cloud-sql-connector/dist/mjs/socket.js:107 - const tlsSocket = tls.connect(socketOpts);
node_modules/.pnpm/@google-cloud+cloud-sql-connector@1.10.0/node_modules/@google-cloud/cloud-sql-connector/dist/mjs/connector.js:148 - const tlsSocket = getSocket({
node_modules/.pnpm/mysql2@3.16.3/node_modules/mysql2/lib/base/connection.js:64 - this.stream = opts.config.stream(opts);
node_modules/.pnpm/mysql2@3.16.3/node_modules/mysql2/lib/base/pool_connection.js:7 - super(options);
node_modules/.pnpm/mysql2@3.16.3/node_modules/mysql2/lib/pool_connection.js:5 - class PoolConnection extends BasePoolConnection {

...

# FILEHANDLE
(unknown stack trace)

# TTYWRAP
temp.direct.mjs:40 - console.log('shutting down while queries are in flight...');

# Timeout
temp.direct.mjs:46 - await new Promise((r) => setTimeout(r, 500));
temp.direct.mjs:46 - await new Promise((r) => setTimeout(r, 500));
```

## How to reproduce

1. `npm init -y && npm pkg set type=module`
2. `npm i @google-cloud/cloud-sql-connector@1.10.0 mysql2@3.16.3 why-is-node-running`
3. Save the example code above as `repro.mjs`, filling in `INSTANCE_CONNECTION_NAME`, `DATABASE`, `USER`.
4. Ensure `gcloud auth application-default login` has been run and the IAM user has access to the instance.
5. `node repro.mjs`
6. Observe `whyIsNodeRunning()` reports N `TLSWRAP` handles rooted at `@google-cloud/cloud-sql-connector/dist/mjs/socket.js` — one per in-flight query that hadn't completed when `pool.end()` was called.

#### Environment details

- OS: MacOS 15.7.4 ARM (M2 Pro)
- Node.js version: v20.20.0 and v24.13.0
- pnpm version: 10.22.0
- `@google-cloud/cloud-sql-connector` version: 1.10.0
- `mysql2` version: 3.16.3

#### Steps to reproduce

1. Create a `Connector` and a `mysql2` pool per the package README.
2. Issue several concurrent queries without awaiting them.
3. Before the queries resolve, call `await pool.end()` and `connector.close()`.
4. Wait long enough (≥ 500 ms) for libuv to collect any handles that were legitimately cleared.
5. Call `whyIsNodeRunning()` (or otherwise inspect the event loop).
6. Expected: no `TLSWRAP` handles from the connector. Actual: one leaked `TLSWRAP` per in-flight query at shutdown time.

Contributor guide

Open the contributing guide

Research direction

Start by reading the connector lifecycle around dist/mjs/socket.js:107 and dist/mjs/connector.js:148, then compare it with mysql2/lib/base/connection.js:64 and the pool connection paths shown in the stack trace. Reproduce the shutdown sequence with concurrent SELECT SLEEP(2) queries and verify that completed shutdown leaves no connector TLSWRAP handles for in-flight queries.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, node.js, typescript
Domain
backend, databases, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.