faced "No such file or directory " when query hudi By Kyuubi 1.7
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Whey query the hudi tables used to get below error messages even after refresh tables:
No such file or directory: s3a://xxxx/3cb781bd-4efc-4277-9b35-6d436f39ddc1-0_24-373672-1708766_20230727133636.parquet
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :0.13.1
* Spark version :3.3.2
* Hive version :3.2
* Hadoop version :3.2.2
* Kyuubi version :1.7.0
* Storage (HDFS/S3/GCS..) : s3
* Running on Docker? (yes/no) :No
I am using Kyuubi integrated with **Hudi 0.13.1** to query hudi tables which is updated by another spark streaming job (spark3.0 ) with **hudi 0.9.0** But used to face below error messages even after refresh tables:
No such file or directory: s3a://xxxx/3cb781bd-4efc-4277-9b35-6d436f39ddc1-0_24-373672-1708766_20230727133636.parquet
Is it because query table with Hudi 0.13.1 but the table is upserted by hudi 0.9.0?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Hudi table query against the reported S3 path with Hudi 0.13.1, Spark 3.3.2, and the table writer using Hudi 0.9.0. Compare the table's file metadata and refresh behavior across those Hudi versions. Done means identifying whether the version combination causes the missing-file error and documenting a confirmed fix or reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, hadoop, java, spark
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100