apache / apache/hudi

Support INSERT SQL statement with a subset of columns in Spark 3.4

Open
#16,805 1 comment 0 reactions 1 assignee Assigned to @yihua View on GitHub
from-jira priority:critical type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

The new tests `TestInsertTable`.`Test Insert Into with subset of columns`, `Test Insert Into with subset of columns on Parquet table` fail on Spark 3.4 due to our validation introduced in HoodieSpark34CatalystPlanUtils in [https://github.com/apache/hudi/pull/11568].  Without this change, INSERT INTO with subset of columns used to work.

 
{code:java}
override def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Seq[String], Map[String, Option[String]], LogicalPlan, Boolean, Boolean)] = {
plan match {
case insert: InsertIntoStatement =>
// https://github.com/apache/spark/pull/36077
// first: in this pr, spark34 support default value for insert into, it will regenerate the user specified cols
// so, no need deal with it in hudi side
// second: in this pr, it will append hoodie meta field with default value, has some bug, it look like be fixed
// in spark35(https://github.com/apache/spark/pull/41262), so if user want specified cols, need disable default feature.
if (SQLConf.get.enableDefaultColumns) {
if (insert.userSpecifiedCols.nonEmpty) {
throw new AnalysisException("hudi not support specified cols when enable default columns, " +
"please disable 'spark.sql.defaultColumn.enabled'")
}
Some((insert.table, Seq.empty, insert.partitionSpec, insert.query, insert.overwrite, insert.ifPartitionNotExists))
} else {
Some((insert.table, insert.userSpecifiedCols, insert.partitionSpec, insert.query, insert.overwrite, insert.ifPartitionNotExists))
}
case _ =>
None
}
} {code}
 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8911
- Type: Bug
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.