apache / apache/gluten

gluten-ut: the flatten function case caches three DataFrames and never unpersists them

Open
#12,858 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### What happens

The Gluten-authored `flatten function` case caches three DataFrames and never releases them. In `gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDataFrameFunctionsSuite.scala` the calls are at `:72`, `:99` and `:116`:

```scala
// Test with cached relation, the Project will be evaluated with codegen
intDF.cache()
```

There is no matching `unpersist()`, and no `try`/`finally` around the cached branch. The same case exists in all five version modules.

### Why it matters

The cached blocks stay in the block manager for the rest of the suite. That is memory pressure the suite does not need, and it makes later cases in the same suite sensitive to execution order: whether a relation is already cached changes the plan they get. Order-dependent test failures are the hard kind to reproduce.

### History

Not new. The case came from `de26ed2da` in `gluten-ut/spark33` and was copied into 3.4/3.5/4.0/4.1 verbatim by #12840, where Copilot flagged it. Whitespace-stripped, the migrated copy is byte-identical to the 3.3 original.

### Suggested fix

Call `unpersist()` after each cached branch's assertions, ideally in a `finally` so a failing assertion still releases the cache. Worth grepping the other `Gluten*Suite` files for the same shape while you are there.

Contributor guide

Open the contributing guide

Research direction

Start with the flatten function case in gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDataFrameFunctionsSuite.scala, at lines 72, 99, and 116, then compare the corresponding cases in the other four version modules. Check the cached branches and their assertions, and grep other Gluten*Suite files for the same shape. Done means every cache is released, including when an assertion fails, and the suite no longer retains those DataFrames.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.