GoogleCloudPlatform / GoogleCloudPlatform/cloud-sql-proxy
Provide performance benchmarks with and without Proxy
- Dominant language
- Go
- Stars
- 1.4k
- Forks
- 352
- Avg merge
- 14h 54m
- Merged PRs (30d)
- 5
Description
### Bug Description
Hello!
I've been investigating some reports from some of our users around latency spikes. I've narrowed the investigation down to when an application receives a large burst of requests, its connection pool can grow quite rapidly, very quickly. What i'm observing is that parallel connections to cloudsql (via cloudsql proxy) seem to have a linear increase in latency with number of connections being made, affecting all connection attempts, not just the first.
I have a test setup that will connect, perform a query, then disconnect.
For example, here's the timings around a single connection attempt:
```
{
"query": 71,
"connect": 70,
"durationMs": 72
}
```
If i make 5 connection attempts in parallel:
```
[
{
"query": 109,
"connect": 108,
"durationMs": 110
},
{
"query": 112,
"connect": 110,
"durationMs": 112
},
{
"query": 118,
"connect": 117,
"durationMs": 119
},
{
"query": 119,
"connect": 116,
"durationMs": 120
},
{
"query": 124,
"connect": 123,
"durationMs": 124
}
]
```
And here is 10:
```
[
{
"query": 147,
"connect": 146,
"durationMs": 147
},
{
"query": 156,
"connect": 155,
"durationMs": 156
},
{
"query": 160,
"connect": 157,
"durationMs": 160
},
{
"query": 161,
"connect": 159,
"durationMs": 161
},
{
"query": 163,
"connect": 161,
"durationMs": 163
},
{
"query": 174,
"connect": 173,
"durationMs": 174
},
{
"query": 180,
"connect": 177,
"durationMs": 180
},
{
"query": 183,
"connect": 179,
"durationMs": 183
},
{
"query": 184,
"connect": 178,
"durationMs": 184
},
{
"query": 185,
"connect": 182,
"durationMs": 185
}
]
```
The pattern continues the more connections I make.
### Example code (or command)
```javascript
This is the crude bit of typescript i wrote to test this:
const timings: { connect: number; query: number; durationMs: number }[] = []
const connectionTest = async (): Promise => {
const start = new Date()
const pgClient = new Client({
host: 'postgres',
port: 5432,
database: 'istio_test',
user: requireEnv('AT_POSTGRES_USERNAME')
})
await pgClient.connect()
const connect = new Date().getTime() - start.getTime()
const results = await pgClient.query('SELECT 1 + 1 AS solution')
if (results.rows[0].solution.toString().trim() !== '2') {
throw new Error('Did not get the expected response from cloudsql')
}
const query = new Date().getTime() - start.getTime()
await pgClient.end()
const end = new Date()
const durationMs = end.getTime() - start.getTime()
timings.push({ query, connect, durationMs })
}
const promises: Promise[] = []
const iterations = parseInt(req.query.iterations ?? '1', 10)
for (let i = 0; i < iterations; i += 1) {
promises.push(connectionTest())
}
await Promise.all(promises)
```
This is using the `pg` library. However all of our users are using java, so this isn't a client library issue.
### Steps to reproduce?
Code sample provided above
### Environment
1. OS type and version: Rocky Linux 8
2. Cloud SQL Proxy version (`./cloud-sql-proxy --version`): 2.5.0
3. Proxy invocation command (for example, `./cloud-sql-proxy --port 5432 INSTANCE_CONNECTION_NAME`):`--private-ip --prometheus --http-address 0.0.0.0 --http-port 9739 --auto-iam-authn, termination period: 30`
### Additional Details
- I have reproduced this locally using a local cloudsql proxy, and gcloud default credentials to connect to the instance.
- I have reproduced this issue without using IAM (username and password instead).
Contributor guide
Assessment
This issue has not been assessed yet.