apache / apache/gluten

[VL] GlutenHiveSQLQuerySuite leaks Hive tables on failure: cleanup sits outside withTable/finally

Open
#12,856 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### What happens

`GlutenHiveSQLQuerySuite` creates a persistent Hive table, runs its assertions, and only then drops it. The drop is a plain statement after the `withSQLConf` block, not inside `withTable` or a `finally`, so any failure in between skips it. Two cases are written this way, and they also mix two cleanup paths: a `DROP TABLE IF EXISTS` at the top of the body and a `spark.sessionState.catalog.dropTable(...)` at the bottom.

- `testGluten("hive orc scan")` (`test_orc`), e.g. `gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLQuerySuite.scala:41` with the drop at `:52`
- `testGluten("avoid unnecessary filter binding for subfield during scan")` (`test_subfield`), same file at `:245` with the drop at `:259`

Both exist in all five `gluten-ut/spark3x` / `spark4x` modules.

### Why it matters

The Hive suites share one `TestHive` singleton, so a leftover table outlives the case that created it. When `checkAnswer` or `checkOperatorMatch` fails, the table stays in the metastore and the warehouse, and later cases in the same JVM see it. The `DROP TABLE IF EXISTS` at the top of the body only rescues a rerun inside the same session, which is not the case CI hits.

### History

Not a new problem. `test_orc` predates the migration in every module. `test_subfield` came from `a662f3f75` (#3300, 2023) and was copied verbatim into 3.4/3.5/4.0/4.1 by #12840, which is where GitHub Copilot flagged it five review rounds in a row. It was left alone there on purpose: that PR was a migration, and rewriting a migrated case would have made it differ from both the 3.3 original and its own neighbour in the same file.

### Suggested fix

Wrap the create/insert/assert work in `withTable("test_orc")` / `withTable("test_subfield")` and delete both manual drops. Do the two cases together across all five modules, otherwise the same file ends up carrying two different cleanup styles, which is worse than the current state.

Contributor guide

Open the contributing guide

Research direction

Start with gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLQuerySuite.scala, then compare the corresponding files in all five spark3x/spark4x modules. Inspect test_orc and test_subfield and run the relevant GlutenHiveSQLQuerySuite tests. Done means both cases use consistent cleanup and no manual drops remain in those five copies.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.