[VL] Velox memCache is invisible to Spark's memory manager
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
When `spark.gluten.sql.columnar.backend.velox.cacheEnabled` is on, `VeloxBackend::initCache()`
builds an `MmapAllocator` sized from `memCacheSize` and hands it to the `AsyncDataCache`:
```cpp
options.capacity = memCacheSize;
cacheAllocator_ = std::make_shared(options);
if (ssdCacheSize == 0) {
// TODO: this is not tracked by Spark.
asyncDataCache_ = velox::cache::AsyncDataCache::create(cacheAllocator_.get());
} else {
// TODO: this is not tracked by Spark.
auto ssd = initSsdCache(ssdCacheSize);
asyncDataCache_ = velox::cache::AsyncDataCache::create(cacheAllocator_.get(), std::move(ssd));
}
```
Spark never sees that memory. The two `// TODO: this is not tracked by Spark.` comments in
`cpp/velox/compute/VeloxBackend.cc` record exactly this.
#### Why it matters
An executor ends up occupying its Spark budget **plus** the whole cache. Every decision Spark
makes from its own books is then made against a number that is short by up to the cache size:
- when to spill,
- when to fail a task with an OOM,
- how much a task may acquire.
The cache is also fixed at its configured size for the lifetime of the executor. It cannot give
anything back when queries need the memory, which is the wrong priority: a query that has to
spill costs far more than the cache misses that shrinking would cause.
#### Proposal
Make the cache capacity something Spark can move at runtime, and register what the cache occupies
with Spark's memory manager, so that:
1. the cache's footprint is visible to Spark on **all** supported Spark versions;
2. the cache yields memory when execution needs it, and reclaims it when the pressure lifts;
3. usage stays bounded by what Spark has agreed to.
An earlier attempt used Spark 4.1's `UnmanagedMemoryConsumer`, which only works on 4.1+ and reacts
only at task boundaries.
This description was written with the assistance of generative AI tooling (GitHub Copilot CLI).
### Gluten version
main branch
Contributor guide
Research direction
Read cpp/velox/compute/VeloxBackend.cc, especially VeloxBackend::initCache(), and trace how the MmapAllocator and AsyncDataCache are created. Done means the cache footprint is visible to Spark on every supported Spark version, can yield and reclaim capacity at runtime, and remains within Spark’s agreed bound.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100