apache / apache/datafusion-comet

[Native Writer] INSERT INTO ... SELECT fails due to stale catalog cache after write

Open
#3,521 1 comment 0 reactions 0 assignees View on GitHub
area:writer bug priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

When using the native Parquet writer with INSERT INTO ... SELECT syntax, subsequent reads return empty results even though the file is written successfully. This could be due to Spark caching .

Ref https://github.com/apache/datafusion-comet/pull/3479/changes#r2804765579

```
withSQLConf(
CometConf.COMET_NATIVE_PARQUET_WRITE_ENABLED.key -> "true",
CometConf.COMET_EXEC_ENABLED.key -> "true",
CometConf.getOperatorAllowIncompatConfigKey(classOf[DataWritingCommandExec]) -> "true") {

sql("create table t(i boolean) using parquet")
sql("alter table t add column s bigint default 42")
sql("insert into t select false, default")
spark.table("t").show() // Returns empty!
}
```

expected : false, 42

actual: (0 rows)

Note: This issue does NOT affect INSERT INTO ... VALUES syntax, only INSERT INTO ... SELECT.

### Steps to reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.