apache / apache/kyuubi

[Bug] Incomplete operation for create table with Delta Lake

Open
#3,056 3 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

After successfully create a delta table with ddl below
```sql
CREATE TABLE IF NOT EXISTS kyuubi_delta (
id INT,
name STRING,
org STRING,
url STRING,
start TIMESTAMP
) USING DELTA
```

Then run a simple query

```sql
select * from kyuubi_delta;
```
Error occurs

```log
org.apache.spark.sql.AnalysisException: `file:/Users/kentyao/incubator-kyuubi/dist/work/kentyao/spark-warehouse/kyuubi_delta` is not a Delta table.
```

Seems that `_delta_log` shall be generated whereas the table location `...spark-warehouse/kyuubi_delta` is empty.

### Affects Version(s)

master

### Kyuubi Server Log Output

_No response_

### Kyuubi Engine Log Output

_No response_

### Kyuubi Server Configurations

_No response_

### Kyuubi Engine Configurations

spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Reproduce the CREATE TABLE and SELECT statements using the listed Spark Delta extensions and catalog configuration, then inspect the generated spark-warehouse/kyuubi_delta location and its _delta_log. Trace the Kyuubi engine path that handles the DDL and verify that the created table can be queried as a Delta table with the expected metadata present.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.