apache / apache/hudi

Fix TestMergeIntoTable2 test

Open
#16,272 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

For the test TestMergeIntoTable2@"Test only insert for source table in dup key without preCombineField", after adding "
spark.sql(s"set ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.key} = 0")", the test fails:
{code:java}
Expected Array([1,a2,10.4,1004,2021-03-21], [1,a2,10.4,1004,2021-03-21], [3,a3,10.3,1003,2021-03-21]), but got Array([1,a2,10.2,1002,2021-03-21], [1,a2,10.4,1004,2021-03-21], [3,a3,10.3,1003,2021-03-21])
ScalaTestFailureLocation: org.apache.spark.sql.hudi.HoodieSparkSqlTestBase at (HoodieSparkSqlTestBase.scala:109)
org.scalatest.exceptions.TestFailedException: Expected Array([1,a2,10.4,1004,2021-03-21], [1,a2,10.4,1004,2021-03-21], [3,a3,10.3,1003,2021-03-21]), but got Array([1,a2,10.2,1002,2021-03-21], [1,a2,10.4,1004,2021-03-21], [3,a3,10.3,1003,2021-03-21])
    at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
    at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
    at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1562)
    at org.scalatest.Assertions.assertResult(Assertions.scala:867)
    at org.scalatest.Assertions.assertResult$(Assertions.scala:863)
    at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1562)
    at org.apache.spark.sql.hudi.HoodieSparkSqlTestBase.checkAnswer(HoodieSparkSqlTestBase.scala:109)
    at org.apache.spark.sql.hudi.TestMergeIntoTable2.$anonfun$new$36(TestMergeIntoTable2.scala:897)
    at org.apache.spark.sql.hudi.TestMergeIntoTable2.$anonfun$new$36$adapted(TestMergeIntoTable2.scala:841)
    at org.apache.spark.sql.hudi.HoodieSparkSqlTestBase.withTempDir(HoodieSparkSqlTestBase.scala:77)
    at org.apache.spark.sql.hudi.TestMergeIntoTable2.$anonfun$new$35(TestMergeIntoTable2.scala:841)
    at org.apache.spark.sql.hudi.TestMergeIntoTable2.$anonfun$new$35$adapted(TestMergeIntoTable2.scala:840)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at org.apache.spark.sql.hudi.TestMergeIntoTable2.$anonfun$new$34(TestMergeIntoTable2.scala:840)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at org.apache.spark.sql.hudi.HoodieSparkSqlTestBase.$anonfun$test$1(HoodieSparkSqlTestBase.scala:85)
    at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
    at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
    at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
    at org.scalatest.Transformer.apply(Transformer.scala:22)
    at org.scalatest.Transformer.apply(Transformer.scala:20)
    at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:189)
    at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
    at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
    at org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1562)
    at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:187)
    at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:199)
    at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
    at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:199)
    at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:181)
    at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1562)
    at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:232)
    at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
    at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
    at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
    at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:232)
    at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:231)
    at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1562)
    at org.scalatest.Suite.run(Suite.scala:1112)
    at org.scalatest.Suite.run$(Suite.scala:1094)
    at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1562)
    at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:236)
    at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
    at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:236)
    at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:235)
    at org.apache.spark.sql.hudi.HoodieSparkSqlTestBase.org$scalatest$BeforeAndAfterAll$$super$run(HoodieSparkSqlTestBase.scala:44)
    at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
    at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
    at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
    at org.apache.spark.sql.hudi.HoodieSparkSqlTestBase.run(HoodieSparkSqlTestBase.scala:44)
    at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
    at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1314)
    at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1308)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1308)
    at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:993)
    at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:971)
    at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1474)
    at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:971)
    at org.scalatest.tools.Runner$.run(Runner.scala:798)
    at org.scalatest.tools.Runner.run(Runner.scala)
    at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:43)
    at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:26) {code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6978
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in TestMergeIntoTable2.scala around lines 841-897, focusing on the TestMergeIntoTable2 case named "Test only insert for source table in dup key without preCombineField" and its MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT setting. Run the test and inspect the checkAnswer failure in HoodieSparkSqlTestBase; done means the test produces the intended result consistently after that setting is added.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
testing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.