DeduplicateKafkaSinkTransformer should work with fields from payload
- Lenguaje dominante
- Scala
- Estrellas
- 47
- Forks
- 14
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
When changing https://github.com/AbsaOSS/hyperdrive/blob/develop/driver/src/test/scala/za/co/absa/hyperdrive/driver/drivers/KafkaToKafkaDeduplicationAfterRetryDockerTest.scala#L73-L74
to
```
"transformer.[kafka.deduplicator].source.id.columns" -> "value.record_id",
"transformer.[kafka.deduplicator].destination.id.columns" -> "value.record_id"
```
the test fails with the following exception:
```
org.apache.spark.SparkException: Malformed records are detected in record parsing.
Caused by: org.apache.avro.AvroTypeException: Found null, expecting string
```
Even if this behavior is documented, it may never have been working.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.