[Bug] Spark 3.3 throws write exceptions during partial-update engine operations
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
master
### Compute Engine
spark 3.3.1
### Minimal reproduce step
create table
```
CREATE TABLE T(
f1 int,
f2 string,
f3 string,
f4 string)
TBLPROPERTIES (
'bucket'='1',
'primary-key'='f1',
'write-only'='true',
'merge-engine'='partial-update',
'file.format'='parquet',
'file.compression' = 'zstd'
)
```
insert
```
INSERT INTO T(f1, f4) VALUES(1, 'test')
```
### What doesn't meet your expectations?
```
Cannot write incompatible data for the table `test.T`, the number of data columns don't match with the table schema's.
java.lang.RuntimeException: Cannot write incompatible data for the table `test.T`, the number of data columns don't match with the table schema's.
at org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis.resolveQueryColumnsByPosition(PaimonAnalysis.scala:135)
at org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis.org$apache$paimon$spark$catalyst$analysis$PaimonAnalysis$$resolveQueryColumns(PaimonAnalysis.scala:78)
at org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis$$anonfun$apply$1.applyOrElse(PaimonAnalysis.scala:43)
at org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis$$anonfun$apply$1.applyOrElse(PaimonAnalysis.scala:41)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$2(AnalysisHelper.scala:170)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$1(AnalysisHelper.scala:170)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning(AnalysisHelper.scala:168)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning$(AnalysisHelper.scala:164)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown(AnalysisHelper.scala:160)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown$(AnalysisHelper.scala:159)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDown(LogicalPlan.scala:30)
at org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis.apply(PaimonAnalysis.scala:41)
at org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis.apply(PaimonAnalysis.scala:39)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:211)
at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
at scala.collection.immutable.List.foldLeft(List.scala:91)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:208)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:200)
```
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with org.apache.paimon.spark.catalyst.analysis.PaimonAnalysis.scala, especially resolveQueryColumnsByPosition at line 135, and reproduce the issue using the partial-update table and INSERT statement in the report. Done means the partial-update insert with only f1 and f4 succeeds without the incompatible-column exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100