Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka
Storage Write API throughput collapses under concurrency: timestamp parsing serializes on the JVM-global TimeZone lock (threeten-bp via google-cloud-bom 26.33.0)
- Vorherrschende Sprache
- Java
- Sterne
- 37
- Forks
- 45
- Ø Merge
- 19 Std. 50 Min.
- Gemergte PRs (30 T.)
- 5
Beschreibung
## Summary
With `useStorageWriteApi: true` (default-stream mode), sink throughput on a high-volume topic collapses far below what the same connector sustains on the legacy insertAll path. Thread dumps show every write thread inside `JsonToProtoMessage.fillField()` -> threeten-bp `DateTimeFormatter.parse()`, serialized on the JVM-global `java.util.TimeZone` class lock (`ZoneTextPrinterParser` path). The per-table `JsonStreamWriter` monitor makes this worse (all `threadPoolSize` threads also funnel through one synchronized `append()`), but removing that lock is not enough: with an experimental build that pools N writers per table, ALL write threads still pile up on the single global TimeZone lock.
This is not specific to that experimental build: dumps from the stock (single-writer) configuration show the same signature -- the thread holding a table's `JsonStreamWriter` monitor is inside the threeten-bp parse (`ZoneTextPrinterParser`), so the TimeZone lock is what makes every hold of the writer monitor long. Removing the writer monitor merely exposed it as the sole remaining serialization point.
This is fixed in newer google-cloud-bigquerystorage releases, where `JsonToProtoMessage` uses lock-free `java.time` instead of threeten-bp. The connector pins google-cloud-bom 26.33.0, which predates that. Proposed fix: raise the BOM floor (we verified 26.83.0 resolves the parse path to `java.time`).
## Environment
- bigquery-connector-for-apache-kafka v2.14.0 (also observed on 2.13.0)
- google-cloud-bom 26.33.0 (as shipped), google-cloud-bigquerystorage 3.2.x
- Kafka Connect 3.8 (Debezium connect image 3.4.3.Final), Java 21
- `useStorageWriteApi: true`, `enableBatchMode: false`, `threadPoolSize: 10`
- Sink topic: Debezium CDC events with several TIMESTAMP fields per record
## Observed behavior (same connector, same topic, same hardware)
- insertAll path: sustains the topic and drains backlog comfortably.
- Storage Write API, single writer per table (stock): fell behind within minutes of enabling, by orders of magnitude more than the producer rate the insertAll path absorbs - plus the issue #200 rebalance cascade when flushes back up.
- Storage Write API, experimental N-writers-per-table build: writer-monitor contention eliminated (zero threads blocked on `SchemaAwareStreamWriter` monitors in dumps), lag growth much slower than single-writer but still strictly behind the producer - the TimeZone lock is the remaining serialization point.
## Thread dump evidence
Captured live under load (20 write threads across 2 sink tasks): 13 threads
BLOCKED on the `TimeZone` class lock, 7 RUNNABLE inside the parse, 0 blocked
on writer monitors, 0 parked in gRPC. Representative stack:
```
"Thread-61" RUNNABLE (holds the parse path)
at org.threeten.bp.format.DateTimeFormatter.parse(DateTimeFormatter.java:1416)
at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.fillField(JsonToProtoMessage.java:597)
at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.convertToProtoMessage(JsonToProtoMessage.java:333)
at com.google.cloud.bigquery.storage.v1.SchemaAwareStreamWriter.buildMessage(SchemaAwareStreamWriter.java:146)
at com.google.cloud.bigquery.storage.v1.SchemaAwareStreamWriter.append(SchemaAwareStreamWriter.java:200)
- locked <0x...> (a com.google.cloud.bigquery.storage.v1.SchemaAwareStreamWriter)
at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:65)
at com.wepay.kafka.connect.bigquery.write.storage.StorageWriteApiDefaultStream$DefaultStreamWriter.appendRows(...)
at com.wepay.kafka.connect.bigquery.write.storage.StorageWriteApiBase.writeBatch(StorageWriteApiBase.java:285)
...
"Thread-32".."Thread-62" (13 of them) BLOCKED
waiting to lock
at org.threeten.bp.zone.... / ZoneTextPrinterParser.parse(...)
at org.threeten.bp.format.DateTimeFormatter.parse(...)
at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.fillField(...)
...
```
I can share fuller (sanitized) dumps if useful.
## Proposed fixes
1. Raise the google-cloud BOM to a version whose `JsonToProtoMessage` uses `java.time` (>= the release that removed threeten-bp from this path; we verified 26.83.0). This is the root fix and benefits every configuration.
2. Separately (companion PR I am happy to open): allow N parallel `JsonStreamWriter` instances per destination table in default-stream mode (`JsonStreamWriter.append()` is synchronized per instance and does the JSON->proto conversion inside the lock, so one writer per table caps effective parallelism at 1 regardless of `threadPoolSize`). The `_default` stream supports concurrent writers and the builder already enables the connection pool, so extra writers cost descriptor state, not connections. Default of 1 = today's behavior. We run this in production.
Related: #200 (a hung AppendRows zombies the task; independent failure mode, but the two compound under load).
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.