Unable to merge cdc records to Hudi snapshot.
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I am unable to add cdc records to a snapshot.
environment
```
emr 7.2.0
AmazonCloudWatchAgent 1.300032.2,
Hive 3.1.3,
Spark 3.5.1,
Zeppelin 0.10.1
```
spark command
```
spark-shell --driver-memory 1g --executor-memory 2500m --executor-cores 1 --driver-cores 1 --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' --conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' --conf 'spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar' --conf "spark.sql.caseSensitive=true" --conf spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED --conf datanucleus.schema.autoCreateTables=true --conf "parquet.avro.write-old-list-structure=false" --packages org.apache.hudi:hudi-spark3.5-bundle_2.12:0.15.0 --name ravic
```
scala code
```
spark.conf.set("parquet.avro.write-old-list-structure",false). // <<----- setting once again here
val sess= spark
var cdcDf = sess.read .schema(snapshotDf.schema) .format("json") .load("s3://bucket/inputs/cdc-jsons/*")
cdcDf.createOrReplaceTempView("cdc")
cdcDf = sess.sql("select * from cdc where _id is not null and _id.oid is not null ")
cdcDf.write.format("hudi")
.options(getQuickstartWriteConfigs)
.option("hoodie.datasource.hive_sync.partition_extractor_class","org.apache.hudi.hive.NonPartitionedExtractor")
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "cdc_pk")
.option("hoodie.datasource.write.recordkey.field", "_id.oid")
.option("parquet.avro.write-old-list-structure","false") // <<---- setting again here ; tried with "false" and false (both)
.option(HoodieWriteConfig.TABLE_NAME, "in_hudi_2")
.mode(SaveMode.Append)
.save("s3://bucket/snapshots-hudi/snapshot/");
```
error
```
Caused by: java.lang.NullPointerException: Array contains a null element at 1. Set parquet.avro.write-old-list-structure=false to turn on support for arrays with null elements.
at org.apache.parquet.avro.AvroWriteSupport$TwoLevelListWriter.writeCollection(AvroWriteSupport.java:606)
at org.apache.parquet.avro.AvroWriteSupport$ListWriter.writeList(AvroWriteSupport.java:444)
at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:372)
at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:291)
at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:204)
at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:178)
at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:138)
at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:310)
at org.apache.hudi.io.hadoop.HoodieBaseParquetWriter.write(HoodieBaseParquetWriter.java:149)
at org.apache.hudi.io.hadoop.HoodieAvroParquetWriter.writeAvro(HoodieAvroParquetWriter.java:80)
at org.apache.hudi.io.storage.HoodieAvroFileWriter.write(HoodieAvroFileWriter.java:51)
at org.apache.hudi.io.storage.HoodieFileWriter.write(HoodieFileWriter.java:43)
at org.apache.hudi.io.HoodieMergeHandle.writeToFile(HoodieMergeHandle.java:395)
at org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:377)
... 41 more
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the supplied Spark 3.5 and Hudi 0.15 setup, then trace the failing write through HoodieMergeHandle.writeToFile and HoodieAvroParquetWriter.writeAvro. Reproduce the append using the provided CDC JSON, snapshot schema, and parquet.avro.write-old-list-structure settings. Done means the null-containing array case has a verified outcome without the reported NullPointerException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100