apache / apache/hudi

[SUPPORT] Missing records when using Kafka Hudi sink to write to S3.

Open
#6,166 7 comments 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:aws engine:kafka-connect issue:data-loss priority:critical release-0.11.0 release-0.14.0
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

I am using the Kafka Hudi sink to write to S3. I am having mismatch in the number messages present in a topic and the number of records showing up in Athena for both MoR and CoW. For MoR, even after running the compaction there are some missing records.

**To Reproduce**

Steps to reproduce the behavior:

1. Initially, I sent 100 messages to a topic. It refelected in Athena after compaction.
2. Later sent 100 more new messages + some updates + some duplicates of previous 100. Record count was not correct.
3. And later sent like 1000 messages and still record count was not correct after compaction.
4. The config file properties are
{
"name": "hudi-sink",
"config": {
"bootstrap.servers": "localhost:9092",
"connector.class": "org.apache.hudi.connect.HoodieSinkConnector",
"tasks.max": "4",
"control.topic.name": "hudi-control-topic-mor",
"topics": "sensor",
"hoodie.table.name": "sensor-mor",
"hoodie.table.type": "MERGE_ON_READ",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"hoodie.base.path": "s3a://path/sensor_mor",
"hoodie.datasource.write.recordkey.field":"oid,styp,sname,ts",
"hoodie.datasource.write.partitionpath.field":"gid,datatype,origin,oid",
"hoodie.datasource.write.keygenerator.type":"COMPLEX",
"hoodie.datasource.write.hive_style_partitioning": "true",
"hoodie.compact.inline.max.delta.commits":2,
"fs.s3a.fast.upload": "true",
"fs.s3a.access.key": "myaccesskey",
"fs.s3a.secret.key": "secretkey",
"hoodie.schemaprovider.class": "org.apache.hudi.schema.SchemaRegistryProvider",
"hoodie.deltastreamer.schemaprovider.registry.url": "http://localhost:8081/subjects/sensor/versions/latest",
"hoodie.kafka.commit.interval.secs": 60
}
}

**Expected behavior**

Irrespective of the messages sent to topic (could be new messages or duplicates or updates), as described, the connector should append them to tables.

**Environment Description**

* Hudi version : 0.11.0

* Spark version : 3.1.3

* Hive version :

* Hadoop version : 3.2

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : No

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.