apache / apache/hudi

[SUPPORT] Cannot load a list of specific files after upgrading to hudi 0.11

Open
#7,176 4 comments 0 reactions 0 assignees View on GitHub
area:aws area:config priority:high release-0.12.2
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

We are using hudi on amazon EMR. We used to use hudi 0.7, which allowed us to use the following syntax to load specific directories of parquet files (in this example just two days):

```python
data_uris = ["s3://bucket/data_product/date=2022-11-01/*", "s3://bucket/data_product/date=2022-11-02/*"]
self.spark.read.format("hudi").option("hoodie.datasource.read.paths", ",".join(data_uris)).load()
```

After upgrading our EMR version we are now using hudi 0.11 but this syntax now fails with:
```
An error was encountered:
An error occurred while calling o409.load.
: org.apache.hudi.exception.HoodieException: 'path' cannot be null, missing 'path' from table properties
```

**To Reproduce**

Steps to reproduce the behavior:

1. Attempt to load two different paths using `spark.read.format('hudi')`

**Expected behavior**

I can load specific directories of data.

**Environment Description**

* Hudi version : 0.11.0

* Spark version : 3.2.1

* Hive version : 3.1.3

* Hadoop version : 3.2.1

* Storage (HDFS/S3/GCS..) : s3

* Running on Docker? (yes/no) : no

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```
An error was encountered:
An error occurred while calling o409.load.
: org.apache.hudi.exception.HoodieException: 'path' cannot be null, missing 'path' from table properties
at org.apache.hudi.Spark3DefaultSource.getTable(Spark3DefaultSource.scala:42)
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:83)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:233)
at scala.Option.map(Option.scala:230)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:750)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at Spark3DefaultSource.scala:42 and reproduce the Spark 3.2.1 read using the two S3 paths shown. Trace why the Hudi source requires a table path, then make loading those specific directories work again and verify that the reported syntax succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.