apache / apache/kyuubi

[Bug] there is no such file or directory

Open
#5,926 6 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

Pre-Description:
1、run sql on spark engine
2、spark engine write and read from hive
3、create hive table as orc or parquet

when i execute sql like “insert overwrite table tpch.ods_tpch_lineitem_d_1 select distinct * from tpch.ods_tpch_lineitem_d;”
i check the hdfs path,find that the kyuubi spark engine will delete the hdfs path and will create the path at the last dag stage
![image](https://github.com/apache/kyuubi/assets/39934213/ba38f9cd-7073-48ec-b60a-92449d770992)

but,there is a issue, if the sql run failed and do not reach the last stage,then the hdfs path of hive table will lost. when i close the spark session and open a new session, run sql “insert overwrite table tpch.ods_tpch_lineitem_d_1 select distinct * from tpch.ods_tpch_lineitem_d;” it will can not find the hdfs path and the task will failed.

### Affects Version(s)

1.7.3/1.8.0

### Kyuubi Server Log Output

_No response_

### Kyuubi Engine Log Output

_No response_

### Kyuubi Server Configurations

_No response_

### Kyuubi Engine Configurations

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or logs are named. Start by reproducing the shown SQL against the Spark engine with Hive-backed ORC or Parquet tables and trace when the HDFS path is removed; done means a failed insert does not leave the table path unavailable to a subsequent session.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, scala, spark, sql
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.