gluten-ut: the flatten function case caches three DataFrames and never unpersists them
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### What happens
The Gluten-authored `flatten function` case caches three DataFrames and never releases them. In `gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDataFrameFunctionsSuite.scala` the calls are at `:72`, `:99` and `:116`:
```scala
// Test with cached relation, the Project will be evaluated with codegen
intDF.cache()
```
There is no matching `unpersist()`, and no `try`/`finally` around the cached branch. The same case exists in all five version modules.
### Why it matters
The cached blocks stay in the block manager for the rest of the suite. That is memory pressure the suite does not need, and it makes later cases in the same suite sensitive to execution order: whether a relation is already cached changes the plan they get. Order-dependent test failures are the hard kind to reproduce.
### History
Not new. The case came from `de26ed2da` in `gluten-ut/spark33` and was copied into 3.4/3.5/4.0/4.1 verbatim by #12840, where Copilot flagged it. Whitespace-stripped, the migrated copy is byte-identical to the 3.3 original.
### Suggested fix
Call `unpersist()` after each cached branch's assertions, ideally in a `finally` so a failing assertion still releases the cache. Worth grepping the other `Gluten*Suite` files for the same shape while you are there.
Contributor guide
Research direction
Start with the flatten function case in gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDataFrameFunctionsSuite.scala, at lines 72, 99, and 116, then compare the corresponding cases in the other four version modules. Check the cached branches and their assertions, and grep other Gluten*Suite files for the same shape. Done means every cache is released, including when an assertion fails, and the suite no longer retains those DataFrames.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100