airbytehq / airbytehq/airbyte

[source-google-analytics-data-api] Silent indefinite freeze on slices >100k rows; paginator stop_condition reads response field GA4 never returns (v2.9.44)

未关闭
#82,766 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
autoteam community connectors/destination/bigquery connectors/source/google-analytics-data-api hyd-review team/use
主要语言
Python
星标
22.1k
派生
5.3k
PR 合并指标
PR 指标待抓取

描述

## Connector Name

source-google-analytics-data-api

## Connector Version

2.9.44

## What step the error happened?

During the sync

## Relevant information

### Summary

On any stream slice whose GA4 `runReport` response exceeds the 100,000-row page limit, the connector **freezes silently and indefinitely**: no records, no state messages, no errors, no retries, and no further API requests. The job stays "running" for hours because the workload heartbeat is orchestrator-level. The freeze is deterministic — we reproduced it 3 times at exactly the same date boundary.

While investigating we also found what looks like a broken paginator definition in `manifest.yaml` (details below) that appears to have shipped with the low-code migration.

### Environment

- Airbyte Cloud
- source-google-analytics-data-api **2.9.44**
- destination-bigquery 3.0.19
- Stream: `ecommerce_purchases_item_id_report` (dimensions `date, itemId`), `window_in_days=1` (default)
- GA4 Standard property that had a bot-traffic surge producing **130k–370k rows/day** for ~3 weeks (normal days: 7k–30k rows, which sync fine)

### Reproduction / evidence

1. Sync starts normally. Days below 100k rows/page complete fine (~700 rec/s).
2. The moment the connector reaches the first day whose report exceeds 100,000 rows (≈45 MB first page), record emission stops mid-sync and never resumes:

```
09:45:31 Syncing stream: ecommerce_purchases_item_id_report
09:45:39 Records read: 5000 (1 MB)
09:45:45 Records read: 10000 (2 MB)
09:45:52 Records read: 15000 (3 MB)
09:45:59 Records read: 20000 (4 MB) <- last record counter ever logged
09:46:12 CheckpointManager: No checkpoints to flush
...("No checkpoints to flush" every 60s forever; no state message is ever emitted)...
```

The destination flushes its buffer 5 minutes later and then idles. The job runs for hours with zero progress until manually cancelled. Because no state is emitted, every retry restarts at the same cursor and freezes at the same day.

3. API-side verification during the freeze (via direct `runReport` calls with `returnPropertyQuota: true` using the same service account): `concurrentRequests: consumed=0`, hourly token quotas ~full. The connector is not waiting on Google and not retrying — the source process is wedged internally. The same >100k-row responses fetch fine outside Airbyte in 3–6 s per 100k-row page when `offset` is passed in the request body.

We suspect the mechanism is memory: with `concurrency_level.default_concurrency: 4` and daily slices, the connector fetches several ~45 MB pages concurrently inside the 2 Gi source container and stalls at the memory ceiling without being OOM-killed. Consistent with this, the freeze always happens at the first contact with the oversized days, right after the last normal-sized day completes.

### Additional bug found while investigating: paginator can never advance

In the current `manifest.yaml` the stream template paginator is:

```yaml
paginator:
type: DefaultPaginator
page_token_option:
type: RequestOption
inject_into: request_parameter # <- query param on a POST-body API
field_name: offset
pagination_strategy:
type: CursorPagination
cursor_value: '{{ response.get("offset", {}) }}'
stop_condition: '{{ not response.get("offset", {}) }}'
```

Two problems:

1. **`stop_condition` reads `response.offset`, but the GA4 Data API `runReport` response never contains an `offset` field** (verified empirically; response keys are `dimensionHeaders, metricHeaders, rows, rowCount, metadata, propertyQuota, kind`). So the stop condition is always true and pagination stops after page 1 — any slice with >100k rows would be silently truncated even if the freeze didn't occur.
2. `offset` is injected as a URL query parameter, but `runReport` expects it in the POST body.

For properties whose daily reports stay under 100k rows this is invisible (single page per slice), which is presumably why it hasn't been widely reported.

### Steps to reproduce

1. Any GA4 property + stream combination where one day's report exceeds 100,000 rows (e.g. `date + itemId` on a high-cardinality catalog).
2. Sync with defaults (`window_in_days=1`).
3. Sync freezes silently at that day; no state committed; repeat cancel/retry freezes at the identical spot.

### Expected behavior

- Slices >100k rows paginate correctly (offset in request body, stop when `rows_fetched >= rowCount`).
- If the source dies/wedges, the attempt should fail visibly instead of hanging indefinitely.

### Workaround we used

Pulled the affected date range directly from the GA4 Data API with body-based offset pagination (works fine, 3–6 s per page) and loaded it into the destination manually; skipped the range in the connection.

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13164

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。