Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Storage Write API throughput collapses under concurrency: timestamp parsing serializes on the JVM-global TimeZone lock (threeten-bp via google-cloud-bom 26.33.0)

未關閉
#238 0 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Java
星號
37
分支
45
平均合併
19 小時 50 分鐘
30 天內合併 PR
5

描述

## Summary

With `useStorageWriteApi: true` (default-stream mode), sink throughput on a high-volume topic collapses far below what the same connector sustains on the legacy insertAll path. Thread dumps show every write thread inside `JsonToProtoMessage.fillField()` -> threeten-bp `DateTimeFormatter.parse()`, serialized on the JVM-global `java.util.TimeZone` class lock (`ZoneTextPrinterParser` path). The per-table `JsonStreamWriter` monitor makes this worse (all `threadPoolSize` threads also funnel through one synchronized `append()`), but removing that lock is not enough: with an experimental build that pools N writers per table, ALL write threads still pile up on the single global TimeZone lock.

This is not specific to that experimental build: dumps from the stock (single-writer) configuration show the same signature -- the thread holding a table's `JsonStreamWriter` monitor is inside the threeten-bp parse (`ZoneTextPrinterParser`), so the TimeZone lock is what makes every hold of the writer monitor long. Removing the writer monitor merely exposed it as the sole remaining serialization point.

This is fixed in newer google-cloud-bigquerystorage releases, where `JsonToProtoMessage` uses lock-free `java.time` instead of threeten-bp. The connector pins google-cloud-bom 26.33.0, which predates that. Proposed fix: raise the BOM floor (we verified 26.83.0 resolves the parse path to `java.time`).

## Environment

- bigquery-connector-for-apache-kafka v2.14.0 (also observed on 2.13.0)
- google-cloud-bom 26.33.0 (as shipped), google-cloud-bigquerystorage 3.2.x
- Kafka Connect 3.8 (Debezium connect image 3.4.3.Final), Java 21
- `useStorageWriteApi: true`, `enableBatchMode: false`, `threadPoolSize: 10`
- Sink topic: Debezium CDC events with several TIMESTAMP fields per record

## Observed behavior (same connector, same topic, same hardware)

- insertAll path: sustains the topic and drains backlog comfortably.
- Storage Write API, single writer per table (stock): fell behind within minutes of enabling, by orders of magnitude more than the producer rate the insertAll path absorbs - plus the issue #200 rebalance cascade when flushes back up.
- Storage Write API, experimental N-writers-per-table build: writer-monitor contention eliminated (zero threads blocked on `SchemaAwareStreamWriter` monitors in dumps), lag growth much slower than single-writer but still strictly behind the producer - the TimeZone lock is the remaining serialization point.

## Thread dump evidence

Captured live under load (20 write threads across 2 sink tasks): 13 threads
BLOCKED on the `TimeZone` class lock, 7 RUNNABLE inside the parse, 0 blocked
on writer monitors, 0 parked in gRPC. Representative stack:

```
"Thread-61" RUNNABLE (holds the parse path)
at org.threeten.bp.format.DateTimeFormatter.parse(DateTimeFormatter.java:1416)
at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.fillField(JsonToProtoMessage.java:597)
at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.convertToProtoMessage(JsonToProtoMessage.java:333)
at com.google.cloud.bigquery.storage.v1.SchemaAwareStreamWriter.buildMessage(SchemaAwareStreamWriter.java:146)
at com.google.cloud.bigquery.storage.v1.SchemaAwareStreamWriter.append(SchemaAwareStreamWriter.java:200)
- locked <0x...> (a com.google.cloud.bigquery.storage.v1.SchemaAwareStreamWriter)
at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:65)
at com.wepay.kafka.connect.bigquery.write.storage.StorageWriteApiDefaultStream$DefaultStreamWriter.appendRows(...)
at com.wepay.kafka.connect.bigquery.write.storage.StorageWriteApiBase.writeBatch(StorageWriteApiBase.java:285)
...

"Thread-32".."Thread-62" (13 of them) BLOCKED
waiting to lock
at org.threeten.bp.zone.... / ZoneTextPrinterParser.parse(...)
at org.threeten.bp.format.DateTimeFormatter.parse(...)
at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.fillField(...)
...
```

I can share fuller (sanitized) dumps if useful.

## Proposed fixes

1. Raise the google-cloud BOM to a version whose `JsonToProtoMessage` uses `java.time` (>= the release that removed threeten-bp from this path; we verified 26.83.0). This is the root fix and benefits every configuration.
2. Separately (companion PR I am happy to open): allow N parallel `JsonStreamWriter` instances per destination table in default-stream mode (`JsonStreamWriter.append()` is synchronized per instance and does the JSON->proto conversion inside the lock, so one writer per table caps effective parallelism at 1 regardless of `threadPoolSize`). The `_default` stream supports concurrent writers and the builder already enables the connection pool, so extra writers cost descriptor state, not connections. Default of 1 = today's behavior. We run this in production.

Related: #200 (a hung AppendRows zombies the task; independent failure mode, but the two compound under load).

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

Start by finding where the project pins google-cloud-bom 26.33.0, then check the dependency tree for google-cloud-bigquerystorage. The issue points to StorageWriteApiBase.writeBatch, JsonStreamWriter.append, and JsonToProtoMessage.fillField as the affected path. Done means the BOM floor is raised to a version such as 26.83.0 and the connector tests/dependency resolution show the newer java.time-based BigQuery Storage library.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
google-cloud, java, kafka
領域
data-engineering, performance
Issue 類型
缺陷
難度
3/5
預估耗時
1-2 天
活躍度
冷清
描述清晰度
基本清楚
新手友好度
55/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。