Aiven-Open / Aiven-Open/transforms-for-apache-kafka-connect

Feature request: Add BigQuery Ingestion Timestamp to Sink Records

Đang mở
#193 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Java
Star
90
Fork
29
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

## Problem

Currently, there is no dedicated column or field in records ingested by the connector that captures the **exact timestamp (in milliseconds) when the record is written/ingested into BigQuery by the Sink Connector**.

The existing `kafkaDataFieldName` configuration creates a bigquery column with `insertTime` but this is the timestamp when the connector writes into the row - not the time that bigquery commits the write.

## Objective
Add a feature to support automatic inclusion of a column/field in each BigQuery row (with millisecond precision) representing the timestamp at which the record is committed to the BigQuery table. This field should be:
- Populated using the worker's wall-clock time in milliseconds immediately before or at the moment of actual ingestion.
- Optionally configurable (e.g., field name, enabled/disabled) in the connector configuration.
- Available regardless of whether the built-in `kafkaDataFieldName` metadata field is enabled

## Benefits
- Debug data unavailability issues
- Measure end-to-end pipeline latency
- Distinguish between Kafka ingestion delay vs BigQuery write delay
- Perform SLA monitoring more accurately

## Workaround
Tables partitioned by ingestion-time have a pseudo column named `_PARTITIONTIME` that stores metadata about the ingestion time for each row. However, the timestamp is truncated at the partition boundary, and will not give you an exact ingestion time.

BigQuery’s `CURRENT_TIMESTAMP()` function returns the current date and time as a timestamp object. However, the value of this function is captured only once at the very start of the query statement, not individually for each row at the point of writing to BigQuery. So while this might reflect the commit time for single-row inserts or very fast statements quite accurately, it becomes increasingly inaccurate as the duration of the INSERT statement increases. Also, when having this as a new column, have to ensure table and record schema are still matched.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.