GoogleCloudPlatform / GoogleCloudPlatform/cloud-sql-go-connector
Provide performance benchmarks with and without Proxy
- Dominant language
- Go
- Stars
- 145
- Forks
- 35
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 5
Description
### Question
Hi there,
We're trying to migrate from RDS to GCP SQL. The application is running on GKE. We're using cloud-sql-go-connector v1.3.0 to connect a Postgres instance like this:
``` golang
cloudSQLDialer, err := cloudsqlconn.NewDialer(ctx, db.WithIAMAuthN(), db.WithDefaultDialOptions(db.WithPrivateIP()))
if err != nil {
panic(errors.Wrap(err, "failed to setup cloud sql dialer"))
}
config.Dialer = func(ctx context.Context, _, _ string) (net.Conn, error) {
span := spans.NewChildSpanFromContext(ctx, "dial-cloudsql") ----------< this span p99 = 300ms
defer span.Finish()
// use context.Background() to avoid cancellation
// canceling context during dial means that connection never makes it to the pool
// so the next query has to pay the price again, potentially hitting timeout as well
//nolint: contextcheck
conn, err := cloudSQLDialer.Dial(context.Background(), connectionName)
if err != nil {
span.SetTag(ext.Error, err)
}
return conn, err
}
```
We noticed that it takes significantly more time for the Dialer to finish ~300ms as opposed to ~30ms for RDS. We assumed this is caused by additional work that the Cloud SQL Proxy server is doing. And indeed we confirmed it by directly connecting to the Cloud SQL Postgres instance using the IP:5432, latency was staying at ~70ms.
Since we do use connection pooling it works with no errors most of the time. But during traffic spikes or after a lot of connections died under heavy load this added latency causes a lot of errors. What is going on there is that when we try to open a significant number of connections at the same time the dialer latency spikes up with every new connection(latency goes up to ~4s), causing the application to try opening even more connections since it can't get enough to fulfill incoming requests. The end result is that the application opens up to the pool limit number of connections(500 in this specific test) and has a lot of errors when it uses cloudsqlconn. And it only opens ~100 connections with no errors when it connects to the IP:5432 directly. Both tests use the same traffic numbers.
Is there anything we can do to mitigate the issue?
Sorry for a lot of text, just want to make sure I give enough info:)
### Code
_No response_
### Additional Details
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.