apache / apache/gluten

[VL][1.2] OOM kill by yarn (aggregate or hash?)

Open
#9,776 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

error msg:
```
ExecutorLostFailure (executor 73 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding physical memory limits. 4.6 GB of 4 GB physical memory used. Consider boosting spark.executor.memoryOverhead.
```

memory profile:

![Image](https://github.com/user-attachments/assets/6cee980e-f213-4b7a-abc1-a55d282d277f)

error plan:
```
{"extensions":[{"extensionFunction":{"functionAnchor":3,"name":"is_not_null:str"}},{"extensionFunction":{"functionAnchor":4,"name":"murmur3hash:opt_i32_str_str_str_str"}},{"extensionFunction":{"functionAnchor":1,"name":"not:bool"}},{"extensionFunction":{"functionAnchor":2,"name":"equal:str_str"}},{"extensionFunction":{"name":"and:bool_bool"}}],"relations":[{"root":{"input":{"project":{"common":{"emit":{"outputMapping":[4,5,6,7,8]}},"input":{"aggregate":{"common":{"direct":{}},"input":{"project":{"common":{"emit":{"outputMapping":[5,6,7,8]}},"input":{"read":{"common":{"direct":{}},"baseSchema":{"names":["id","p1","update_time","id2","dt"],"struct":{"types":[{"string":{"nullability":"NULLABILITY_NULLABLE"}},{"string":{"nullability":"NULLABILITY_NULLABLE"}},{"string":{"nullability":"NULLABILITY_NULLABLE"}},{"string":{"nullability":"NULLABILITY_NULLABLE"}},{"string":{"nullability":"NULLABILITY_NULLABLE"}}]},"columnTypes":["NORMAL_COL","NORMAL_COL","NORMAL_COL","NORMAL_COL","PARTITION_COL"]},"filter":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":1,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":2,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{"field":3}}}}},{"value":{"literal":{"string":"0"}}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":1,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":2,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{"field":3}}}}},{"value":{"literal":{"string":"NULL"}}}]}}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":3,"outputType":{"bool":{"nullability":"NULLABILITY_REQUIRED"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{"field":3}}}}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":1,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":2,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{}}}}},{"value":{"literal":{"string":"NULL"}}}]}}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":3,"outputType":{"bool":{"nullability":"NULLABILITY_REQUIRED"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{}}}}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":1,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":2,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{}}}}},{"value":{"literal":{"string":"000000000000000"}}}]}}}]}}}]}}},{"value":{"singularOrList":{"value":{"selection":{"directReference":{"structField":{"field":1}}}},"options":[{"literal":{"string":"1"}},{"literal":{"string":"2"}},{"literal":{"string":"9"}},{"literal":{"string":"10"}},{"literal":{"string":"11"}},{"literal":{"string":"22"}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":1,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":2,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{"field":3}}}}},{"value":{"literal":{"string":""}}}]}}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":1,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":2,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{}}}}},{"value":{"literal":{"string":""}}}]}}}]}}}]}},"advancedExtension":{"optimization":{"@type":"/google.protobuf.StringValue","value":"isMergeTree=0\n"}}}},"expressions":[{"selection":{"directReference":{"structField":{}}}},{"selection":{"directReference":{"structField":{"field":1}}}},{"selection":{"directReference":{"structField":{"field":2}}}},{"selection":{"directReference":{"structField":{"field":3}}}}]}},"groupings":[{"groupingExpressions":[{"selection":{"directReference":{"structField":{"field":3}}}},{"selection":{"directReference":{"structField":{}}}},{"selection":{"directReference":{"structField":{"field":1}}}},{"selection":{"directReference":{"structField":{"field":2}}}}]}],"advancedExtension":{"optimization":{"@type":"/google.protobuf.StringValue","value":"isStreaming=0\nallowFlush=1\n"}}}},"expressions":[{"scalarFunction":{"functionReference":4,"outputType":{"i32":{"nullability":"NULLABILITY_REQUIRED"}},"arguments":[{"value":{"literal":{"i32":42}}},{"value":{"selection":{"directReference":{"structField":{}}}}},{"value":{"selection":{"directReference":{"structField":{"field":1}}}}},{"value":{"selection":{"directReference":{"structField":{"field":2}}}}},{"value":{"selection":{"directReference":{"structField":{"field":3}}}}}]}},{"selection":{"directReference":{"structField":{}}}},{"selection":{"directReference":{"structField":{"field":1}}}},{"selection":{"directReference":{"structField":{"field":2}}}},{"selection":{"directReference":{"structField":{"field":3}}}}]}},"names":["hash_partition_key#279","id2#61","id#49","p1#51","update_time#55","hash_partition_key#279","id2#61","id#49","p1#51","update_time#55"]}}]}
```

![Image](https://github.com/user-attachments/assets/af38c86c-a458-477e-aeae-c9793be03d81)

### Gluten version

Gluten-1.2

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start with the supplied YARN ExecutorLostFailure, memory profile, and error plan, focusing on the aggregate and hash stages in the Velox backend. Reproduce the workload if possible and isolate whether aggregation or hash partitioning drives the physical-memory limit; done means the responsible stage and a verified fix or mitigation are identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
backend, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.