AbsaOSS / AbsaOSS/hyperdrive

DeduplicateKafkaSinkTransformer should work with fields from payload

未关闭
#239 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Scala
星标
47
派生
14
PR 合并指标
30 天内没有已合并 PR

描述

When changing https://github.com/AbsaOSS/hyperdrive/blob/develop/driver/src/test/scala/za/co/absa/hyperdrive/driver/drivers/KafkaToKafkaDeduplicationAfterRetryDockerTest.scala#L73-L74

to
```
"transformer.[kafka.deduplicator].source.id.columns" -> "value.record_id",
"transformer.[kafka.deduplicator].destination.id.columns" -> "value.record_id"
```
the test fails with the following exception:
```
org.apache.spark.SparkException: Malformed records are detected in record parsing.
Caused by: org.apache.avro.AvroTypeException: Found null, expecting string
```

Even if this behavior is documented, it may never have been working.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。