firebase / firebase/firebase-tools

Data Connect: bursty load at half the documented 24k req/min limit fails with opaque "SQL execution failed"

Open
#11,068 0 comments 3 reactions 0 assignees View on GitHub
api: dataconnect type: bug
Dominant language
TypeScript
Stars
4.5k
Forks
1.3k
Avg merge
1d 12h
Merged PRs (30d)
84

Description

### [REQUIRED] Environment info

**firebase-tools:** 15.22.1

**Platform:** Cloud Run (Node 22, `firebase-admin` 13.10.0), Data Connect service in `europe-west1` backed by Cloud SQL for PostgreSQL 17 (`db-custom-8-16384`)

We could not reproduce the issue on db-custom-1-3840

### [REQUIRED] Test case

When Dataconnect is under pressure, we start getting
```
SQL execution failed
at DataConnectApiClient.makeGqlRequest (/usr/src/app/typescript/node_modules/firebase-admin/lib/data-connect/data-connect-api-client-internal.js:270:19)
```
on random queries.

The error does not include any details about quota limits, etc. It just says "SQL execution failed".

In order to get more info about the error, we tried the following:

Standard Admin SDK usage through a generated connector SDK — one shared `DataConnect` instance for the whole process:

```ts
import { DataConnect, getDataConnect } from 'firebase-admin/data-connect'
import { connectorConfig } from '@firebasegen/backend-connector'

const dataConnect: DataConnect = getDataConnect({
connector: connectorConfig.connector,
serviceId: connectorConfig.serviceId,
location: connectorConfig.location,
})
// generated SDK operations, e.g.:
// await sdk.touchContent(dataConnect, { contentId })
```

The mutation itself is trivial (single-row `_update`):

```graphql
mutation TouchContent($contentId: String!) {
content_update(id: $contentId, data: { updated_expr: "request.time" })
}
```

Minimal load generator (plain REST against the connector, which is how we captured the full error `extensions` that the Admin SDK drops):

```js
import { GoogleAuth } from 'google-auth-library'

const URL = 'https://firebasedataconnect.googleapis.com/v1/projects//locations/europe-west1/services//connectors/:impersonateMutation'
const auth = new GoogleAuth({ scopes: ['https://www.googleapis.com/auth/cloud-platform'] })
const client = await auth.getClient()

const CONCURRENCY = 200

// each worker keeps exactly one request in flight, so total in-flight = CONCURRENCY
async function worker() {
while (true) {
const { token } = await client.getAccessToken()
const res = await fetch(URL, {
method: 'POST',
headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' },
body: JSON.stringify({ operationName: 'TouchContent', variables: { contentId: '' } }),
})
const body = await res.json()
if (body.errors?.length) console.error(JSON.stringify(body.errors))
}
}
await Promise.all(Array.from({ length: CONCURRENCY }, () => worker()))
```

### [REQUIRED] Steps to reproduce

1. Run the load generator above at concurrency 200. Each mutation takes ~1s server-side, so throughput settles around **8,000–12,000 operations/min — half of the documented limit** of 24,000 GraphQL requests per minute per service ([docs](https://firebase.google.com/docs/sql-connect/manage-services-and-databases)).
2. Let the concurrency ramp up quickly from idle
3. Within the first seconds/minutes of the burst, operations fail. The same failure signature occurs in production every time bursty load hits, while the Cloud SQL instance is demonstrably healthy: 17% CPU, 43% memory, ~140 write IOPS vs ~1,100 provisioned, zero errors in `postgres.log`.

### [REQUIRED] Expected behavior

- Staying under the only documented limit (24,000 GraphQL requests/min) means operations don't fail due to internal capacity limits.
- If Data Connect's own infrastructure is throttled, the error is surfaced with a **retryable** code (`UNAVAILABLE` / `RESOURCE_EXHAUSTED`), not `INTERNAL`.
- `firebase-admin` surfaces the GraphQL error `extensions` so the cause is diagnosable. Today `DataConnectApiClient.makeGqlRequest` throws only the generic message and discards `extensions.debugDetails` — the only part of the response that explains what happened.

### [REQUIRED] Actual behavior

The **only** signal we get is:

```
SQL execution failed Error: SQL execution failed
at DataConnectApiClient.makeGqlRequest (.../firebase-admin/lib/data-connect/data-connect-api-client-internal.js:270:19)
at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
at async DataConnectApiClient.executeOperationHelper (.../firebase-admin/lib/data-connect/data-connect-api-client-internal.js:178:26)
```

No quota information. Calling the connector REST endpoint directly reveals the real cause in `extensions.debugDetails`:

```json
{"message":"SQL execution failed","path":["content_update"],"extensions":{"code":"INTERNAL","debugDetails":"SQL runtime error: MyStubby connect error to instance \":europe-west1:\": cannot receive first message: ZERO_APP::1: stream aborted: server handler returned error: ERROR_NOT_AUTHORIZED: GetConnectSettings request failed on instance / with error: boss::260: Quota exceeded for quota metric 'Connect Queries' and limit 'Connect Queries per minute per user per region' of service 'sqladmin.googleapis.com' for consumer 'project_number:'. ..."}}
```

Note the consumer: `project_number:` is **not our project** (neither our dev nor prod project number) — it appears to be a Google-managed tenant project where Data Connect's connection brokering runs. So this quota is invisible in our console/monitoring and cannot be raised by us.

What Cloud Monitoring on our side shows (metric `cloudsql.googleapis.com/database/postgresql/new_connection_count`, 1-min sum):

- Baseline: 8–20 new connections/min.
- During every failure burst the metric jumps to a plateau of **~2.000 new connections/min** — exactly the shape of a 2,000/min quota ceiling. This correlates 1:1 with the error bursts in both our dev repro and production.

So Data Connect appears to open and close Cloud SQL connections at a very high rate under bursty load, each new connection paying `GetConnectSettings`/ephemeral-cert calls against the Cloud SQL Admin API `Connect Queries` quota (default 2,000/min per user per region) — and that internal quota, not anything documented or visible to us, is what fails our operations.

Please note that the issue did not reproduce when postgres db tier was db-custom-1-3840 and it only reproduced when upgrading to db-custom-8-16384

**Asks:**
1. Fix the connection churn (pool/reuse connections across bursts) or size the internal quota for the service's own behavior — or document the real limit ("max new connections per minute per service") so it can be designed around.
2. Return a retryable error code instead of `INTERNAL` / generic `SQL execution failed` for this failure mode.
3. Expose GraphQL error `extensions` on the thrown error

Contributor guide

Open the contributing guide

Research direction

Start with DataConnectApiClient.makeGqlRequest in firebase-admin/lib/data-connect/data-connect-api-client-internal.js and compare its handling with the connector REST response. Run the provided concurrency-200 load generator and inspect the GraphQL extensions.debugDetails and Cloud SQL connection metrics. Done means the quota failure is diagnosable, classified as retryable when appropriate, and the relevant error details are preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, graphql, node.js, postgresql, typescript
Domain
api, backend, cloud, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.