apache / apache/hudi

[SUPPORT] Can't redefine array

Open
#11,807 6 comments 0 reactions 0 assignees View on GitHub
area:schema engine:spark
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Dear community,
I found an error in using Hudi. If we use the array> type in the table.Afterwards, every time I add columns and then write , it throws an exception, The error message is as follows:
image
Has anyone encountered this issue? How should it be resolved?

Steps to reproduce the behavior:
1.create table
CREATE TABLE IF NOT EXISTS db.table (
`p_id` bigint ,
`record_items` array> ,
`edit_items` array> ,
`texts` array> ,
`p_date` string
) using hudi
PARTITIONED BY (p_date)
options (
type = 'mor',
primaryKey = 'p_id',
payloadClass='org.apache.hudi.common.model.PartialUpdateAvroPayload',

'hoodie.bucket.index.num.buckets'='4'
);
2.insert into table db.table
select
130928195078 as p_id,
array(named_struct("name", "John", "value", 10),
named_struct("name", "wangwu", "value", 5),
named_struct("name", "Bob", "value", 7)) as record_items,
array(named_struct("name", "zhangsan", "value", 10),
named_struct("name", "Jane", "value", 5),
named_struct("name", "lisi", "value", 7)) as edit_items,
null as texts,
'20240810' as p_date
;
3.alter table db.table add columns(is_new_col string);
4.insert into table db.table
select
130928195078 as p_id,
array(named_struct("name", "John", "value", 10),
named_struct("name", "wangwu", "value", 5),
named_struct("name", "Bob", "value", 7)) as record_items,
array(named_struct("name", "zhangsan", "value", 10),
named_struct("name", "Jane", "value", 5),
named_struct("name", "lisi", "value", 7)) as edit_items,
null as texts,
'newcol' as is_new_col,
'20240810' as p_date
;

**Environment Description**

* Hudi version :0.14.0

* Spark version :2.4

* Hadoop version :2.6

* Storage (HDFS/S3/GCS..) :HDFS

stack
org.apache.spark.SparkException: org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:394)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:394)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleInsertPartition(BaseSparkCommitActionExecutor.java:400)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$2(BaseSparkCommitActionExecutor.java:310)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$25.apply(RDD.scala:873)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$25.apply(RDD.scala:873)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:344)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:308)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:344)
at org.apache.spark.rdd.RDD$$anonfun$7.apply(RDD.scala:357)
at org.apache.spark.rdd.RDD$$anonfun$7.apply(RDD.scala:355)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1184)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1158)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1093)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1158)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:884)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:355)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:306)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:344)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:308)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:95)
at org.apache.spark.scheduler.Task.run(Task.scala:124)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:495)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1388)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:959)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.avro.SchemaParseException: Can't redefine: array
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:149)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdateInternal(BaseSparkCommitActionExecutor.java:439)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdate(BaseSparkCommitActionExecutor.java:421)
at org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:80)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:387)
... 31 more
Caused by: org.apache.avro.SchemaParseException: Can't redefine: array
at org.apache.avro.Schema$Names.put(Schema.java:1128)
at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:562)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:690)
at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:805)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:882)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:716)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:701)
at org.apache.avro.Schema.toString(Schema.java:324)
at org.apache.avro.Schema.toString(Schema.java:314)
at org.apache.parquet.avro.AvroReadSupport.setAvroReadSchema(AvroReadSupport.java:69)
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getIndexedRecordIteratorInternal(HoodieAvroParquetReader.java:162)
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getIndexedRecordIterator(HoodieAvroParquetReader.java:94)
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getRecordIterator(HoodieAvroParquetReader.java:73)
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:126)
... 35 more

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the SQL sequence in the issue on Hudi 0.14.0 with Spark 2.4, then trace the failure from HoodieMergeHelper.java into HoodieAvroParquetReader.java and the reported Avro schema stack. Done means adding a column and performing the second insert no longer fails with SchemaParseException: Can't redefine: array.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, spark
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.