[SUPPORT] Partial Update with Partition column is not working as expected.
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Partial Update with some partition column/key is not working as expected.**
Let's suppose I currently have the following event/message.
```
{
"id": 1,
"language": "python",
"created": "2023-07-12",
"updated": "2023-07-12"
}
```
**primaryKey** = id
**deDupKey/preCombine** = updated
**partition** = created
I am applying UPSERT as a writeOperation type.
Now I want to apply the partial update when receiving a record from my source system/producer.
The new incoming event is as follows.
```
{
"id": 1,
"language": "scala",
"updated": "2023-07-13"
}
```
Now after a partial update, I want to update only columns like language & updated column. But after applying the partial update, we are getting null in the CREATED column.
The expected result after the merge/partial update should be
```
{
"id": 1,
"language": "scala",
"created": "2023-07-12",
"updated": "2023-07-13"
}
```
But it is coming as
```
{
"id": 1,
"language": "scala",
"created": null,
"updated": "2023-07-13"
}
```
Which is actually wrong. Will you please help us here? Are we doing something wrong?
**Environment Description**
Hudi version : 0.13.1
Spark version: 3.1
Hive version: 3.1
Storage (HDFS/S3/GCS..) : GCS
Running on Docker? (yes/no) : No, running on Dataproc
**Hudi Configs**
```
val hudiConfigs: Map[String, String] = Map(
"hoodie.datasource.write.hive_style_partitioning" -> "true",
"hoodie.datasource.write.drop.partition.columns" -> "true",
"hoodie.partition.metafile.use.base.format" -> "true",
"hoodie.metadata.enable" -> "true",
"hoodie.datasource.write.reconcile.schema" -> "true",
"hoodie.schema.on.read.enable" -> "true",
"hoodie.upsert.shuffle.parallelism" -> "1000",
"hoodie.bloom.index.parallelism" -> "1000",
"hoodie.index.type" -> "GLOBAL_BLOOM",
"hoodie.datasource.write.payload.class" -> "org.apache.hudi.common.model.PartialUpdateAvroPayload"
)
```
**To Reproduce**
Steps to reproduce the behaviour:
1. Initiate the spark session & pass the hudi configs mentioned above
2. Choose ID as the primary Key, created a partition column & updated as deDup/preCombine field.
3. First insert the record by supplying all the columns.
4. In a partial update, don't pass the created column, pass the schema which will make the `created` column as null.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the partial-update scenario with the supplied Spark, Hudi 0.13.1, partitioning, and payload configuration, focusing on org.apache.hudi.common.model.PartialUpdateAvroPayload. Compare the record before and after the update; done means the omitted created partition value remains 2023-07-12 while language and updated change as expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100