Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka
Issue on "Date Format" using the S3 Sink Connector from Avro to Parquet
- Langage dominant
- Java
- Étoiles
- 58
- Forks
- 39
- Merge moyen
- 2 j 10 h
- PR mergées (30 j)
- 5
Description
Hi, when using the connector to share event-data from Posgresql Debezium to S3 via Kafka in Parquet, we have an issue to get a "Date Format"
In Kafka, the payload is :
` "created_date": 1643631507020, `
The schema created by the Debezium is this one
```
{
"name": "created_date",
"type": {
"type": "long",
"connect.version": 1,
"connect.name": "org.apache.kafka.connect.data.Timestamp",
"logicalType": "timestamp-millis"
}
},
```
Using a S3 connector to share this data as "Parquet file", we can configure a smt transformation to transform as string
` "transforms.TsCreatedDate.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"transforms.TsCreatedDate.field": "created_date",
"transforms.TsCreatedDate.format": "yyyy-MM-dd'T'HH:mm:ssZ",
"transforms.TsCreatedDate.target.type": "string",
`
But the expected date format in Parquet is **date**.
We still get the "long format" or "string" with SMT transformation.
` required int64 created_date;`
Attended format should be: DATE, TIMESTAMP_MILLIS, TIMESTAMP_MICROS
How can we resolve this ?
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.