apache / apache/hudi

faced "No such file or directory " when query hudi By Kyuubi 1.7

Open
#9,487 1 comment 0 reactions 0 assignees View on GitHub
component:kyuubi priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Whey query the hudi tables used to get below error messages even after refresh tables:

No such file or directory: s3a://xxxx/3cb781bd-4efc-4277-9b35-6d436f39ddc1-0_24-373672-1708766_20230727133636.parquet


A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version :0.13.1

* Spark version :3.3.2

* Hive version :3.2

* Hadoop version :3.2.2

* Kyuubi version :1.7.0

* Storage (HDFS/S3/GCS..) : s3

* Running on Docker? (yes/no) :No

I am using Kyuubi integrated with **Hudi 0.13.1** to query hudi tables which is updated by another spark streaming job (spark3.0 ) with **hudi 0.9.0** But used to face below error messages even after refresh tables:

No such file or directory: s3a://xxxx/3cb781bd-4efc-4277-9b35-6d436f39ddc1-0_24-373672-1708766_20230727133636.parquet

Is it because query table with Hudi 0.13.1 but the table is upserted by hudi 0.9.0?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Hudi table query against the reported S3 path with Hudi 0.13.1, Spark 3.3.2, and the table writer using Hudi 0.9.0. Compare the table's file metadata and refresh behavior across those Hudi versions. Done means identifying whether the version combination causes the missing-file error and documenting a confirmed fix or reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, hadoop, java, spark
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.