Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

BigQuerySinkTask.preCommit() hangs indefinitely when Storage Write API gRPC call never completes

Abierto
#200 4 comentarios 1 reacción 0 asignados Ver en GitHub
Lenguaje dominante
Java
Estrellas
37
Forks
45
Merge medio
19 h 50 min
PR fusionados (30 d)
5

Descripción

## Summary
`BigQuerySinkTask.preCommit()` → `flush()` → `KcbqThreadPoolExecutor.awaitCurrentTasks()`
blocks on `CountDownLatch.await()` with no timeout. If a Storage Write API `AppendRows`
gRPC call never completes (observed after `ConnectionWorker.resetConnection()` during
stream pool scaling), the write task never counts down the latch. `flush()` waits
forever. The Kafka Connect task thread is zombied until the consumer group fences it
(`max.poll.interval.ms`), triggering a rebalance. The task restarts, makes progress
for a few minutes, and hangs again. Repeats indefinitely.

## Environment
- bigquery-connector 2.13.0
- Kafka Connect 3.8 (Debezium connect image 3.4.3.Final)
- GCP BigQuery Storage Write API
- `useStorageWriteApi: true`, `enableBatchMode: false`

## Thread dump
"task-thread-batch-production-bigquery-sink-stream-2" — WAITING (parking)
at jdk.internal.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007251724d0> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
at com.wepay.kafka.connect.bigquery.write.batch.KcbqThreadPoolExecutor.awaitCurrentTasks(KcbqThreadPoolExecutor.java:99)
at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.flush(BigQuerySinkTask.java:191)
at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.preCommit(BigQuerySinkTask.java:211)
at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:445)

Gax/gRPC threads are all idle (`waiting on condition` in `ThreadPoolExecutor.getTask()`),
confirming no in-flight gRPC work — the latch is waiting on a write task that will
never complete.

Preceded in the logs by:
com.google.cloud.bigquery.storage.v1.ConnectionWorkerPool createConnectionWorker
INFO: Scaling up new connection for stream name: .../batch_requests/_default, pool size after scaling up 1
com.google.cloud.bigquery.storage.v1.ConnectionWorker resetConnection

## Impact
Consumer lag grows unboundedly. Observed ~1.4B offset lag over 3 days. 94 rebalances
in 13 hours from the zombie→fence→rebalance cycle.

## Workaround
`useStorageWriteApi: false` — the legacy streaming inserts HTTP path has timeouts,
so failed write tasks complete (with an exception) and the latch counts down.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.