apache / apache/hudi

[SUPPORT] Upgrade from 0.6.0 to 0.15.0

Open
#11,738 5 comments 0 reactions 0 assignees View on GitHub
area:upgrade
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

We have much hudi table of version 0.6.0, We want to upgrade to version 0.14.1 or 0.15.0, so we did some test. When we write table of version 0.6.0 with client of 0.15.0, some error happen

**To Reproduce**

Steps to reproduce the behavior:

1. create table with 0.6.0
2. write table with 0.15.0
3.
4.

**Expected behavior**

upgrade success

**Environment Description**

* Hudi version : 0.6.0 & 0.15.0

* Spark version :

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) :

* Running on Docker? (yes/no) :

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```scala
Exception in thread "main" org.apache.hudi.exception.HoodieException: Config conflict(key current value existing value):
RecordKey: id null
at org.apache.hudi.HoodieWriterUtils$.validateTableConfig(HoodieWriterUtils.scala:229)
HoodieWriterUtils.scala:229
at org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:232)
HoodieSparkSqlWriter.scala:232
at org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:187)
HoodieSparkSqlWriter.scala:187
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:125)
HoodieSparkSqlWriter.scala:125
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:168)
DefaultSource.scala:168
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47)
SaveIntoDataSourceCommand.scala:47
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
commands.scala:75
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
commands.scala:73
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
commands.scala:84
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:97)
QueryExecution.scala:97
```

the reason is table of 0.6.0 not record fields info in hoodie.properties
```
hoodie.table.precombine.field=x
hoodie.table.partition.fields=y
hoodie.table.recordkey.fields=x
```

but when write with client of 0.15.0, it did some validation, the error occur in the validation. the validation should skip when current table version is too low without need info in hoodie.properties

```scala
object HoodieSparkSqlWriter {

private def writeInternal(sqlContext: SQLContext,
mode: SaveMode,
optParams: Map[String, String],
...) {

var tableConfig = getHoodieTableConfig(sparkContext, path, mode, ...)

// 验证option参数和hoodie.properties是否一致
// 低版本的hoodie.properties没有记录字段信息
validateTableConfig(sqlContext.sparkSession, optParams, tableConfig, mode == SaveMode.Overwrite)
}
}
```

when we fill fields info in hoodie.properties of version 0.6.0 by ourself, the upgrade success.

by the way: is it safe upgrade from 0.6.0 to 0.15.0 or 0.14.1 ?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading HoodieWriterUtils.scala at the reported validateTableConfig failure and the table-version/config handling in HoodieSparkSqlWriter.scala. Check how validation behaves for 0.6.0 tables whose hoodie.properties omit record-field information, and look for related tests. Done means an upgrade write no longer fails on missing legacy config while preserving validation for newer tables; the issue is old and has had no recent response.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.