apache / apache/hudi

[FEATURE REQUEST] Support union field

Open
#12,778 1 comment 2 reactions 0 assignees View on GitHub
area:schema
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

Currently this is not possible to write Hudi data if the source DF contains a field with union type.

Let's say you have an avro schema with an union type field :
```json
{
"name": "field",
"type":
[
"null",
"int",
"string",
"boolean"
],
"default": null
}
```

If you create a parquet file from this the field will be transformed into a struct with a memberX for each type of the union `field>`

Then when writing data in Hudi, in 0.X, the process fails because it will take only the first type and then try to write the struct into the type selected, in our case you will get something like :
`java.lang.IllegalArgumentException: StructType(StructField(member0,IntegerType,true),StructField(member1,StringType,true),StructField(member2,BooleanType,true)) and IntegerType are incompatible:`

This behavior was a bit [opaque](https://github.com/apache/hudi/blob/release-0.14.1/hudi-common/src/main/java/org/apache/hudi/internal/schema/convert/AvroInternalSchemaConverter.java#L199-L204) in 0.x but in 1.0.0 this has been made pretty clear [here
](https://github.com/apache/hudi/blob/14c292c626cd8d18b5997a90cfbb865befb5f6d2/hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaUtils.java#L436-L439)

So each time we encounter parquet files containing fields with union type we need to pre-process the data as a inelegant workaround (renaming the `memberX` fields to avoid the union type detection)

Knowing that the parquet implementation allows this union type and Avro as well, we could expect Hudi to be able to handle it in one or other way (representing the member struct as it is ?). Wdyt ?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the union-handling logic in hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaUtils.java and the related AvroInternalSchemaConverter.java code linked in the issue. Reproduce the failure with the provided union schema and Parquet-shaped struct, then determine and document the intended Hudi representation before adding coverage for successful writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.