apache / apache/paimon

[Bug] Spark 3.3 throws write exceptions during partial-update engine operations

Open
#5,932 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

master

### Compute Engine

spark 3.3.1

### Minimal reproduce step

create table
```
CREATE TABLE T(
f1 int,
f2 string,
f3 string,
f4 string)
TBLPROPERTIES (
'bucket'='1',
'primary-key'='f1',
'write-only'='true',
'merge-engine'='partial-update',
'file.format'='parquet',
'file.compression' = 'zstd'
)
```
insert
```
INSERT INTO T(f1, f4) VALUES(1, 'test')
```

### What doesn't meet your expectations?

```
Cannot write incompatible data for the table `test.T`, the number of data columns don't match with the table schema's.
java.lang.RuntimeException: Cannot write incompatible data for the table `test.T`, the number of data columns don't match with the table schema's.
at org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis.resolveQueryColumnsByPosition(PaimonAnalysis.scala:135)
at org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis.org$apache$paimon$spark$catalyst$analysis$PaimonAnalysis$$resolveQueryColumns(PaimonAnalysis.scala:78)
at org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis$$anonfun$apply$1.applyOrElse(PaimonAnalysis.scala:43)
at org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis$$anonfun$apply$1.applyOrElse(PaimonAnalysis.scala:41)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$2(AnalysisHelper.scala:170)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$1(AnalysisHelper.scala:170)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning(AnalysisHelper.scala:168)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning$(AnalysisHelper.scala:164)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown(AnalysisHelper.scala:160)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown$(AnalysisHelper.scala:159)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDown(LogicalPlan.scala:30)
at org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis.apply(PaimonAnalysis.scala:41)
at org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis.apply(PaimonAnalysis.scala:39)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:211)
at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
at scala.collection.immutable.List.foldLeft(List.scala:91)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:208)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:200)
```

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis.scala, especially resolveQueryColumnsByPosition at line 135, and reproduce the issue using the partial-update table and INSERT statement in the report. Done means the partial-update insert with only f1 and f4 succeeds without the incompatible-column exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.