GoogleCloudPlatform / GoogleCloudPlatform/cloud-sql-proxy

Provide performance benchmarks with and without Proxy

Open
#1,871 14 comments 0 reactions 1 assignee Assigned to @hessjcg View on GitHub
priority: p2 type: feature request
Dominant language
Go
Stars
1.4k
Forks
352
Avg merge
14h 54m
Merged PRs (30d)
5

Description

### Bug Description

Hello!
I've been investigating some reports from some of our users around latency spikes. I've narrowed the investigation down to when an application receives a large burst of requests, its connection pool can grow quite rapidly, very quickly. What i'm observing is that parallel connections to cloudsql (via cloudsql proxy) seem to have a linear increase in latency with number of connections being made, affecting all connection attempts, not just the first.

I have a test setup that will connect, perform a query, then disconnect.

For example, here's the timings around a single connection attempt:
```
{
"query": 71,
"connect": 70,
"durationMs": 72
}
```

If i make 5 connection attempts in parallel:

```
[
{
"query": 109,
"connect": 108,
"durationMs": 110
},
{
"query": 112,
"connect": 110,
"durationMs": 112
},
{
"query": 118,
"connect": 117,
"durationMs": 119
},
{
"query": 119,
"connect": 116,
"durationMs": 120
},
{
"query": 124,
"connect": 123,
"durationMs": 124
}
]
```

And here is 10:

```
[
{
"query": 147,
"connect": 146,
"durationMs": 147
},
{
"query": 156,
"connect": 155,
"durationMs": 156
},
{
"query": 160,
"connect": 157,
"durationMs": 160
},
{
"query": 161,
"connect": 159,
"durationMs": 161
},
{
"query": 163,
"connect": 161,
"durationMs": 163
},
{
"query": 174,
"connect": 173,
"durationMs": 174
},
{
"query": 180,
"connect": 177,
"durationMs": 180
},
{
"query": 183,
"connect": 179,
"durationMs": 183
},
{
"query": 184,
"connect": 178,
"durationMs": 184
},
{
"query": 185,
"connect": 182,
"durationMs": 185
}
]
```

The pattern continues the more connections I make.

### Example code (or command)

```javascript
This is the crude bit of typescript i wrote to test this:

const timings: { connect: number; query: number; durationMs: number }[] = []
const connectionTest = async (): Promise => {
const start = new Date()
const pgClient = new Client({
host: 'postgres',
port: 5432,
database: 'istio_test',
user: requireEnv('AT_POSTGRES_USERNAME')
})
await pgClient.connect()
const connect = new Date().getTime() - start.getTime()
const results = await pgClient.query('SELECT 1 + 1 AS solution')
if (results.rows[0].solution.toString().trim() !== '2') {
throw new Error('Did not get the expected response from cloudsql')
}
const query = new Date().getTime() - start.getTime()
await pgClient.end()
const end = new Date()
const durationMs = end.getTime() - start.getTime()
timings.push({ query, connect, durationMs })
}

const promises: Promise[] = []

const iterations = parseInt(req.query.iterations ?? '1', 10)
for (let i = 0; i < iterations; i += 1) {
promises.push(connectionTest())
}
await Promise.all(promises)
```

This is using the `pg` library. However all of our users are using java, so this isn't a client library issue.

### Steps to reproduce?

Code sample provided above

### Environment

1. OS type and version: Rocky Linux 8
2. Cloud SQL Proxy version (`./cloud-sql-proxy --version`): 2.5.0
3. Proxy invocation command (for example, `./cloud-sql-proxy --port 5432 INSTANCE_CONNECTION_NAME`):`--private-ip --prometheus --http-address 0.0.0.0 --http-port 9739 --auto-iam-authn, termination period: 30`

### Additional Details

- I have reproduced this locally using a local cloudsql proxy, and gcloud default credentials to connect to the instance.
- I have reproduced this issue without using IAM (username and password instead).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.