[VL] Task killed by Yarn
Open
bug
tracker
triage
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
one common issue of Gluten is that the task is killed by yarn. Currently Gluten has some memory allocation like std::vector isn't tracked by spark's memory management. It's counted into executor.overhead memory now. But there should be some such allocation with large data size. We should create a proxy allocator for such memory allocation.
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.