Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka

Issue on "Date Format" using the S3 Sink Connector from Avro to Parquet

Ouverte
#280 1 commentaire 2 réactions 0 personnes assignées Voir sur GitHub
S3
Langage dominant
Java
Étoiles
58
Forks
39
Merge moyen
2 j 10 h
PR mergées (30 j)
5

Description

Hi, when using the connector to share event-data from Posgresql Debezium to S3 via Kafka in Parquet, we have an issue to get a "Date Format"

In Kafka, the payload is :
` "created_date": 1643631507020, `

The schema created by the Debezium is this one
```
{
"name": "created_date",
"type": {
"type": "long",
"connect.version": 1,
"connect.name": "org.apache.kafka.connect.data.Timestamp",
"logicalType": "timestamp-millis"
}
},
```

Using a S3 connector to share this data as "Parquet file", we can configure a smt transformation to transform as string
` "transforms.TsCreatedDate.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"transforms.TsCreatedDate.field": "created_date",
"transforms.TsCreatedDate.format": "yyyy-MM-dd'T'HH:mm:ssZ",
"transforms.TsCreatedDate.target.type": "string",
`

But the expected date format in Parquet is **date**.
We still get the "long format" or "string" with SMT transformation.
` required int64 created_date;`

Attended format should be: DATE, TIMESTAMP_MILLIS, TIMESTAMP_MICROS

How can we resolve this ?

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.