[SUPPORT] Cannot load a list of specific files after upgrading to hudi 0.11
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
We are using hudi on amazon EMR. We used to use hudi 0.7, which allowed us to use the following syntax to load specific directories of parquet files (in this example just two days):
```python
data_uris = ["s3://bucket/data_product/date=2022-11-01/*", "s3://bucket/data_product/date=2022-11-02/*"]
self.spark.read.format("hudi").option("hoodie.datasource.read.paths", ",".join(data_uris)).load()
```
After upgrading our EMR version we are now using hudi 0.11 but this syntax now fails with:
```
An error was encountered:
An error occurred while calling o409.load.
: org.apache.hudi.exception.HoodieException: 'path' cannot be null, missing 'path' from table properties
```
**To Reproduce**
Steps to reproduce the behavior:
1. Attempt to load two different paths using `spark.read.format('hudi')`
**Expected behavior**
I can load specific directories of data.
**Environment Description**
* Hudi version : 0.11.0
* Spark version : 3.2.1
* Hive version : 3.1.3
* Hadoop version : 3.2.1
* Storage (HDFS/S3/GCS..) : s3
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
An error was encountered:
An error occurred while calling o409.load.
: org.apache.hudi.exception.HoodieException: 'path' cannot be null, missing 'path' from table properties
at org.apache.hudi.Spark3DefaultSource.getTable(Spark3DefaultSource.scala:42)
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:83)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:233)
at scala.Option.map(Option.scala:230)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:750)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at Spark3DefaultSource.scala:42 and reproduce the Spark 3.2.1 read using the two S3 paths shown. Trace why the Hudi source requires a table path, then make loading those specific directories work again and verify that the reported syntax succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100