apache / apache/hudi

[SUPPORT] COW+hiveStylePartitioning+glob.paths on Spark: reads incomplete values of partition column

Open
#11,856 2 comments 0 reactions 0 assignees View on GitHub
area:schema issue:data-corruption
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

When Spark reads `COW` table with `hiveStylePartitioning` with `glob.paths` (`path` contains '*' or `hoodie.datasource.read.paths` is set) WITH partition values contains '/' - partition values are incomplete (only the first part is read as the value of column).
If table is MOR OR partitioning is not hive style OR wildcard is not used while loading OR partition values don't contain slashes - everything works correct.

Do you have any ideas how to fix it? Or may be Hudi has nothing to do with this because it's something wrong on Spark side?

**To Reproduce**

1. create COW table with hive_style_partitioning;
2. insert some records with '/' in values of partition column ('2024/08/29') into this table;
3. load data from this table using wildcard (`load(s"$basePath/*/*/*")`);
4. show or print the loaded data.

Here is the test case (`extends HoodieSparkSqlTestBase`):
```
test("Test partition incomplete") {
withTempDir { tmp =>
val tbName = "wk_date"
val basePath = s"$tmp/$tbName"
val columns = Seq("id","driver","precomb","dat")
val data = Seq((1,"driver-A",6,"2021/01/01"),(2,"driver-B",7,"2021/01/02"),(3,"driver-C",8,"2021/03/01"))

val inserts = spark.createDataFrame(data).toDF(columns:_*)

val hudi_options = Map(
"hoodie.table.name" -> tbName,
"hoodie.datasource.write.table.type"-> "COPY_ON_WRITE",
"hoodie.datasource.write.recordkey.field" -> "id",
"hoodie.datasource.write.precombine.field" -> "precomb",
"hoodie.datasource.write.partitionpath.field" -> "dat",
"hoodie.datasource.write.hive_style_partitioning" -> "true"
)
inserts.write.format("hudi").options(hudi_options)
.mode(org.apache.spark.sql.SaveMode.Overwrite).save(basePath)

val df = spark.read.format("hudi").load(s"$basePath/*/*/*")
df.select((Seq("_hoodie_partition_path") ++ columns).map(c => col(c)):_*).show()
}
}
```
And here is it's output (only year of `dat` values instead of full existing dates):
```
+----------------------+---+--------+-------+----+
|_hoodie_partition_path| id| driver|precomb| dat|
+----------------------+---+--------+-------+----+
| dat=2021/03/01| 3|driver-C| 8|2021|
| dat=2021/01/02| 2|driver-B| 7|2021|
| dat=2021/01/01| 1|driver-A| 6|2021|
+----------------------+---+--------+-------+----+
```
And directory structure is as follows:
```
dat=2021
|- 01
|- 01
|- 02
|- 03
|- 01
```

**Expected behavior**

Values in partition column of loaded data must be the same as were written (2021/03/01, 2021/01/02, 2021/01/01).

**Environment Description**

* Hudi version : 0.15 and actual master

* Spark version : 3.3, 3.5

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) :

* Running on Docker? (yes/no) : no

**Additional context**

I've found two tickets in Jira that may be somehow related to this issue:
https://issues.apache.org/jira/browse/HUDI-7484 - Fix partitioning style when partition is inferred from partitionBy
https://issues.apache.org/jira/browse/HUDI-7724 - Deprecate usage of `glob.path` to simplify read path in Spark
But there aren't any details on how to solve them.

**Stacktrace**

No errors, just incorrect values in partition column.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the provided HoodieSparkSqlTestBase reproducer and its spark.read.format("hudi").load wildcard path, comparing the written partition value with the loaded column value. Done means the test preserves complete values such as 2021/03/01 for COW hive-style partitions containing slashes, while the other stated cases continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.