[VL] Flaky test from Spark-40 and Spark-41
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
```
- SPARK-32388: TRANSFORM should handle schema less correctly (no serde)
17:20:17.096 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: LocalTableScan, due to: Gluten does not touch it or does not support it
17:20:17.096 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: SparkScriptTransformation, due to: Gluten does not touch it or does not support it
E20260727 17:20:17.165253 7538 VeloxMemoryManager.cc:446] Failed to release Velox memory manager as there are still outstanding memory resources.
E20260727 17:20:17.165344 7538 Exceptions.h:55] Line: /work/ep/build-velox/build/velox_ep/velox/common/memory/Memory.cpp:196, Function:~MemoryManager, Expression: pools_.size() != 0 (1 vs 0). There are unexpected alive memory pools allocated by user on memory manager destruction:
Memory Manager[capacity UNLIMITED alignment 64B usedBytes 0B number of pools 4
List of root pools:
__sys_root__ usage 0B reserved 0B peak 0B
__sys_shared_leaf__0 usage 0B reserved 0B peak 0B
__sys_tracing__ usage 0B reserved 0B peak 0B
__sys_caching__ usage 0B reserved 0B peak 0B
__sys_spilling__ usage 0B reserved 0B peak 0B
root usage 0B reserved 0B peak 0B
default_leaf usage 0B reserved 0B peak 0B
refcount 2
Memory Allocator[MALLOC capacity UNLIMITED allocated bytes 0 allocated pages 0 mapped pages 0]
ARBITRATOR[GLUTEN] CAPACITY 8388608.00TB numRequests 0 numRunning 0 numSucceded 0 numAborted 0 numFailures 0 numNonReclaimableAttempts 0 reclaimedFreeCapacity 0B reclaimedUsedCapacity 0B maxCapacity 0B freeCapacity 0B freeReservedCapacity 0B], Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
what(): Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: pools_.size() != 0 (1 vs 0). There are unexpected alive memory pools allocated by user on memory manager destruction:
Memory Manager[capacity UNLIMITED alignment 64B usedBytes 0B number of pools 4
List of root pools:
__sys_root__ usage 0B reserved 0B peak 0B
__sys_shared_leaf__0 usage 0B reserved 0B peak 0B
__sys_tracing__ usage 0B reserved 0B peak 0B
__sys_caching__ usage 0B reserved 0B peak 0B
__sys_spilling__ usage 0B reserved 0B peak 0B
17:20:17.163 ERROR org.apache.spark.sql.execution.SparkScriptTransformationExec: /bin/bash: some_non_existent_command: command not found
```
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start by locating the Spark-40 and Spark-41 tests and reproducing the reported failure around SPARK-32388. Compare the test teardown with the Velox memory-manager error and SparkScriptTransformation log; done means the tests no longer fail intermittently and complete without outstanding memory pools.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100