apache / apache/gluten

[VL] Flaky test from Spark-40 and Spark-41

Open
#12,649 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

```
- SPARK-32388: TRANSFORM should handle schema less correctly (no serde)
17:20:17.096 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: LocalTableScan, due to: Gluten does not touch it or does not support it

17:20:17.096 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: SparkScriptTransformation, due to: Gluten does not touch it or does not support it

E20260727 17:20:17.165253 7538 VeloxMemoryManager.cc:446] Failed to release Velox memory manager as there are still outstanding memory resources.
E20260727 17:20:17.165344 7538 Exceptions.h:55] Line: /work/ep/build-velox/build/velox_ep/velox/common/memory/Memory.cpp:196, Function:~MemoryManager, Expression: pools_.size() != 0 (1 vs 0). There are unexpected alive memory pools allocated by user on memory manager destruction:
Memory Manager[capacity UNLIMITED alignment 64B usedBytes 0B number of pools 4
List of root pools:
__sys_root__ usage 0B reserved 0B peak 0B
__sys_shared_leaf__0 usage 0B reserved 0B peak 0B
__sys_tracing__ usage 0B reserved 0B peak 0B
__sys_caching__ usage 0B reserved 0B peak 0B
__sys_spilling__ usage 0B reserved 0B peak 0B

root usage 0B reserved 0B peak 0B
default_leaf usage 0B reserved 0B peak 0B

refcount 2
Memory Allocator[MALLOC capacity UNLIMITED allocated bytes 0 allocated pages 0 mapped pages 0]
ARBITRATOR[GLUTEN] CAPACITY 8388608.00TB numRequests 0 numRunning 0 numSucceded 0 numAborted 0 numFailures 0 numNonReclaimableAttempts 0 reclaimedFreeCapacity 0B reclaimedUsedCapacity 0B maxCapacity 0B freeCapacity 0B freeReservedCapacity 0B], Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
what(): Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: pools_.size() != 0 (1 vs 0). There are unexpected alive memory pools allocated by user on memory manager destruction:
Memory Manager[capacity UNLIMITED alignment 64B usedBytes 0B number of pools 4
List of root pools:
__sys_root__ usage 0B reserved 0B peak 0B
__sys_shared_leaf__0 usage 0B reserved 0B peak 0B
__sys_tracing__ usage 0B reserved 0B peak 0B
__sys_caching__ usage 0B reserved 0B peak 0B
__sys_spilling__ usage 0B reserved 0B peak 0B
17:20:17.163 ERROR org.apache.spark.sql.execution.SparkScriptTransformationExec: /bin/bash: some_non_existent_command: command not found

```

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by locating the Spark-40 and Spark-41 tests and reproducing the reported failure around SPARK-32388. Compare the test teardown with the Velox memory-manager error and SparkScriptTransformation log; done means the tests no longer fail intermittently and complete without outstanding memory pools.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.