apache / apache/gluten

[VL] Remove explicit use of utility methods `ensureLoaded()` and `ensureOffloaded()`

Open
#540 0 comments 0 reactions 0 assignees View on GitHub
enhancement stale velox backend
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

A couple of developers might discover that it's not that easy to decide when to use `ensureLoaded` to load native data to JVM as well as `ensureOffloaded`. The existing of the methods were to implement lazy load/offload on data to reduce the overall JNI overhead running queires.

**Describe the solution you'd like**
Spark 3.3 made class `ColumnarBatch` inheritable. We can implement the class by (e.g.) `GlutenLoadedColumnarBatch` and `GlutenOffloadedColumnarBatch`, then we don't need developers to explicitly use `ensureLoaded` and `ensureOffloaded`.

This is a placeholder issue since we are now still on Spark 3.2 and `ColumnarBatch` is still not allowed to be extended.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.