apache / apache/hudi

[SUPPORT] mergedSchema behavior when reading multiple hudi tables at once

Open
#9,498 2 comments 0 reactions 0 assignees View on GitHub
priority:medium type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Hello,

I'm using Hudi 0.11 on Spark 3.2.1 on EMR 6.7.0. I have a ingestion pipeline where data is written out in daily batches and each batch is its own HUDI table. In order words, data is structured like so:

```
basePath/
2023-08-10/ /* this is a HUDI table */
partitionColumn1/
partitionColumn2/
2023-08-11/ /* this is another HUDI table */
partitionColumn1/
partitionColumn2/
...
```

The schema is common across all tables. We recently added a column to this schema and were trying to figure out the best way of handling this across out downstream data processing jobs, which will typically read in a date range of data of the above table.

I found a solution utilizing the `mergeSchema` option. If I read all tables in like so, then my data is read in with the correct updated schema:

```
def getHudiReadOptions(s3ReadPath: String): Map[String, String] = Map(
DataSourceReadOptions.QUERY_TYPE.key() -> DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL,
DataSourceReadOptions.READ_PATHS.key() -> s3ReadPath,
HoodieMetadataConfig.ENABLE.key() -> "true"
)

val paths = List(
"s3://daily-data/2023-08-14/*/*/*", /* this path has the old schema */
"s3://daily-data/2023-08-15/*/*/*", /* this path has the new field */
)
val merged = spark.read.format("org.apache.hudi")
.options(getHudiReadOptions(paths.mkString(",")))
.option("mergeSchema", "true")
.load()
```

If I do not include the `mergeSchema` option, then the data is read in but is missing the new field. My question is: is this expected behavior? Can we rely on the mergeSchema option to handle these kinds of schema differences? I have read through the schema evolution documentation listed here (https://hudi.apache.org/docs/0.11.0/schema_evolution) but have not seen a mention of using this option. Our use case is not typical with multiple hudi tables per day of data, so wanted to check to ensure that this behavior is reliable.

**Environment Description**

* Hudi version : 0.11

* Spark version : 3.2.1

* Hive version : 3.1.3

* Hadoop version : 3.2.1

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : No

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Spark read entry point shown in the issue, including getHudiReadOptions and the mergeSchema option, then compare the behavior against Hudi's 0.11 schema evolution documentation. Confirm whether reading the listed daily table paths with and without mergeSchema is supported and reliable, and document the expected result for the reported schema difference.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, scala
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.