apache / apache/gluten

[VL] table scan allocated more memory than used and caused OOM

Open
#11,876 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

It's a very simple query:
```sql
select max(ss_wholesale_cost),max(ss_list_price)
from store_sales
where ss_quantity between 1 and 20
```
table scan reports OOM while the system memory utilization is only 64% and has 22G free memory available

```
org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::addInput failed for [operator: PartialAggregation, plan node ID: 2]: Error during calling Java code from native code: org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8.0 MiB, granted: 2.1 MiB. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
spark.memory.offHeap.enabled=True
spark.gluten.memory.dynamic.offHeap.sizing.enabled=N/A
spark.gluten.memory.offHeap.size.in.bytes=17.2 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=1098.1 MiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=549.1 MiB
Memory consumer stats:
Task.96: Current used bytes: 1096.0 MiB, peak bytes: N/A
\- Gluten.Tree.95: Current used bytes: 1096.0 MiB, peak bytes: 1098.1 MiB
\- Capacity[8.0 EiB].94: Current used bytes: 1096.0 MiB, peak bytes: 1098.1 MiB
+- NativePlanEvaluator-93.0: Current used bytes: 1096.0 MiB, peak bytes: 1098.1 MiB
| \- single: Current used bytes: 1096.0 MiB, peak bytes: 1096.0 MiB
| +- root: Current used bytes: 1088.0 MiB, peak bytes: 1089.0 MiB
| | +- task.Gluten_Stage_3_TID_96_VTID_93: Current used bytes: 1088.0 MiB, peak bytes: 1089.0 MiB
| | | +- node.0: Current used bytes: 1087.9 MiB, peak bytes: 1088.0 MiB
| | | | +- op.0.0.0.TableScan: Current used bytes: 1087.9 MiB, peak bytes: 1088.0 MiB
| | | | \- op.0.0.0.TableScan.test-hive: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | +- node.2: Current used bytes: 64.4 KiB, peak bytes: 1024.0 KiB
| | | | \- op.2.0.0.PartialAggregation: Current used bytes: 64.4 KiB, peak bytes: 64.4 KiB
| | | \- node.1: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.1.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- default: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- ShuffleWriter.93: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- VeloxBatchResizer.97.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- IteratorMetrics.92: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- NativePlanEvaluator-93.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- ShuffleWriter.93.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- IteratorMetrics.92.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- VeloxBatchResizer.97: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- default: Current used bytes: 0.0 B, peak bytes: 0.0 B

```

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported SQL query and reviewing the table-scan and PartialAggregation memory report, including the Spark off-heap settings. Compare allocated memory with actual usage and verify that the query completes without an erroneous OOM under the same conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
backend, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.