[VL] GlutenHiveSQLQuerySuite leaks Hive tables on failure: cleanup sits outside withTable/finally
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### What happens
`GlutenHiveSQLQuerySuite` creates a persistent Hive table, runs its assertions, and only then drops it. The drop is a plain statement after the `withSQLConf` block, not inside `withTable` or a `finally`, so any failure in between skips it. Two cases are written this way, and they also mix two cleanup paths: a `DROP TABLE IF EXISTS` at the top of the body and a `spark.sessionState.catalog.dropTable(...)` at the bottom.
- `testGluten("hive orc scan")` (`test_orc`), e.g. `gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLQuerySuite.scala:41` with the drop at `:52`
- `testGluten("avoid unnecessary filter binding for subfield during scan")` (`test_subfield`), same file at `:245` with the drop at `:259`
Both exist in all five `gluten-ut/spark3x` / `spark4x` modules.
### Why it matters
The Hive suites share one `TestHive` singleton, so a leftover table outlives the case that created it. When `checkAnswer` or `checkOperatorMatch` fails, the table stays in the metastore and the warehouse, and later cases in the same JVM see it. The `DROP TABLE IF EXISTS` at the top of the body only rescues a rerun inside the same session, which is not the case CI hits.
### History
Not a new problem. `test_orc` predates the migration in every module. `test_subfield` came from `a662f3f75` (#3300, 2023) and was copied verbatim into 3.4/3.5/4.0/4.1 by #12840, which is where GitHub Copilot flagged it five review rounds in a row. It was left alone there on purpose: that PR was a migration, and rewriting a migrated case would have made it differ from both the 3.3 original and its own neighbour in the same file.
### Suggested fix
Wrap the create/insert/assert work in `withTable("test_orc")` / `withTable("test_subfield")` and delete both manual drops. Do the two cases together across all five modules, otherwise the same file ends up carrying two different cleanup styles, which is worse than the current state.
Contributor guide
Research direction
Start with gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLQuerySuite.scala, then compare the corresponding files in all five spark3x/spark4x modules. Inspect test_orc and test_subfield and run the relevant GlutenHiveSQLQuerySuite tests. Done means both cases use consistent cleanup and no manual drops remain in those five copies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- databases, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100