apache / apache/hudi

Unable to merge cdc records to Hudi snapshot.

Open
#11,785 6 comments 0 reactions 0 assignees View on GitHub
area:writer
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I am unable to add cdc records to a snapshot.

environment
```
emr 7.2.0
AmazonCloudWatchAgent 1.300032.2,
Hive 3.1.3,
Spark 3.5.1,
Zeppelin 0.10.1
```

spark command
```
spark-shell --driver-memory 1g --executor-memory 2500m --executor-cores 1 --driver-cores 1 --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' --conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' --conf 'spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar' --conf "spark.sql.caseSensitive=true" --conf spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED --conf datanucleus.schema.autoCreateTables=true --conf "parquet.avro.write-old-list-structure=false" --packages org.apache.hudi:hudi-spark3.5-bundle_2.12:0.15.0 --name ravic

```

scala code
```
spark.conf.set("parquet.avro.write-old-list-structure",false). // <<----- setting once again here

val sess= spark
var cdcDf = sess.read .schema(snapshotDf.schema) .format("json") .load("s3://bucket/inputs/cdc-jsons/*")
cdcDf.createOrReplaceTempView("cdc")
cdcDf = sess.sql("select * from cdc where _id is not null and _id.oid is not null ")
cdcDf.write.format("hudi")
.options(getQuickstartWriteConfigs)
.option("hoodie.datasource.hive_sync.partition_extractor_class","org.apache.hudi.hive.NonPartitionedExtractor")
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "cdc_pk")
.option("hoodie.datasource.write.recordkey.field", "_id.oid")

.option("parquet.avro.write-old-list-structure","false") // <<---- setting again here ; tried with "false" and false (both)

.option(HoodieWriteConfig.TABLE_NAME, "in_hudi_2")
.mode(SaveMode.Append)
.save("s3://bucket/snapshots-hudi/snapshot/");

```
error
```
Caused by: java.lang.NullPointerException: Array contains a null element at 1. Set parquet.avro.write-old-list-structure=false to turn on support for arrays with null elements.
at org.apache.parquet.avro.AvroWriteSupport$TwoLevelListWriter.writeCollection(AvroWriteSupport.java:606)
at org.apache.parquet.avro.AvroWriteSupport$ListWriter.writeList(AvroWriteSupport.java:444)
at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:372)
at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:291)
at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:204)
at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:178)
at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:138)
at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:310)
at org.apache.hudi.io.hadoop.HoodieBaseParquetWriter.write(HoodieBaseParquetWriter.java:149)
at org.apache.hudi.io.hadoop.HoodieAvroParquetWriter.writeAvro(HoodieAvroParquetWriter.java:80)
at org.apache.hudi.io.storage.HoodieAvroFileWriter.write(HoodieAvroFileWriter.java:51)
at org.apache.hudi.io.storage.HoodieFileWriter.write(HoodieFileWriter.java:43)
at org.apache.hudi.io.HoodieMergeHandle.writeToFile(HoodieMergeHandle.java:395)
at org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:377)
... 41 more

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the supplied Spark 3.5 and Hudi 0.15 setup, then trace the failing write through HoodieMergeHandle.writeToFile and HoodieAvroParquetWriter.writeAvro. Reproduce the append using the provided CDC JSON, snapshot schema, and parquet.avro.write-old-list-structure settings. Done means the null-containing array case has a verified outcome without the reported NullPointerException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.