apache / apache/hudi

[SUPPORT] Partial Update with Partition column is not working as expected.

Open
#9,217 7 comments 0 reactions 0 assignees View on GitHub
component:partial-update priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Partial Update with some partition column/key is not working as expected.**

Let's suppose I currently have the following event/message.

```
{
"id": 1,
"language": "python",
"created": "2023-07-12",
"updated": "2023-07-12"
}
```

**primaryKey** = id
**deDupKey/preCombine** = updated
**partition** = created

I am applying UPSERT as a writeOperation type.

Now I want to apply the partial update when receiving a record from my source system/producer.

The new incoming event is as follows.

```
{
"id": 1,
"language": "scala",
"updated": "2023-07-13"
}
```

Now after a partial update, I want to update only columns like language & updated column. But after applying the partial update, we are getting null in the CREATED column.

The expected result after the merge/partial update should be

```
{
"id": 1,
"language": "scala",
"created": "2023-07-12",
"updated": "2023-07-13"
}
```

But it is coming as

```
{
"id": 1,
"language": "scala",
"created": null,
"updated": "2023-07-13"
}
```

Which is actually wrong. Will you please help us here? Are we doing something wrong?

**Environment Description**

Hudi version : 0.13.1
Spark version: 3.1
Hive version: 3.1
Storage (HDFS/S3/GCS..) : GCS
Running on Docker? (yes/no) : No, running on Dataproc

**Hudi Configs**

```
val hudiConfigs: Map[String, String] = Map(
"hoodie.datasource.write.hive_style_partitioning" -> "true",
"hoodie.datasource.write.drop.partition.columns" -> "true",
"hoodie.partition.metafile.use.base.format" -> "true",
"hoodie.metadata.enable" -> "true",
"hoodie.datasource.write.reconcile.schema" -> "true",
"hoodie.schema.on.read.enable" -> "true",
"hoodie.upsert.shuffle.parallelism" -> "1000",
"hoodie.bloom.index.parallelism" -> "1000",
"hoodie.index.type" -> "GLOBAL_BLOOM",
"hoodie.datasource.write.payload.class" -> "org.apache.hudi.common.model.PartialUpdateAvroPayload"
)
```

**To Reproduce**

Steps to reproduce the behaviour:

1. Initiate the spark session & pass the hudi configs mentioned above
2. Choose ID as the primary Key, created a partition column & updated as deDup/preCombine field.
3. First insert the record by supplying all the columns.
4. In a partial update, don't pass the created column, pass the schema which will make the `created` column as null.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the partial-update scenario with the supplied Spark, Hudi 0.13.1, partitioning, and payload configuration, focusing on org.apache.hudi.common.model.PartialUpdateAvroPayload. Compare the record before and after the update; done means the omitted created partition value remains 2023-07-12 while language and updated change as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.