[VL][1.2] OOM kill by yarn (aggregate or hash?)
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
error msg:
```
ExecutorLostFailure (executor 73 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding physical memory limits. 4.6 GB of 4 GB physical memory used. Consider boosting spark.executor.memoryOverhead.
```
memory profile:

error plan:
```
{"extensions":[{"extensionFunction":{"functionAnchor":3,"name":"is_not_null:str"}},{"extensionFunction":{"functionAnchor":4,"name":"murmur3hash:opt_i32_str_str_str_str"}},{"extensionFunction":{"functionAnchor":1,"name":"not:bool"}},{"extensionFunction":{"functionAnchor":2,"name":"equal:str_str"}},{"extensionFunction":{"name":"and:bool_bool"}}],"relations":[{"root":{"input":{"project":{"common":{"emit":{"outputMapping":[4,5,6,7,8]}},"input":{"aggregate":{"common":{"direct":{}},"input":{"project":{"common":{"emit":{"outputMapping":[5,6,7,8]}},"input":{"read":{"common":{"direct":{}},"baseSchema":{"names":["id","p1","update_time","id2","dt"],"struct":{"types":[{"string":{"nullability":"NULLABILITY_NULLABLE"}},{"string":{"nullability":"NULLABILITY_NULLABLE"}},{"string":{"nullability":"NULLABILITY_NULLABLE"}},{"string":{"nullability":"NULLABILITY_NULLABLE"}},{"string":{"nullability":"NULLABILITY_NULLABLE"}}]},"columnTypes":["NORMAL_COL","NORMAL_COL","NORMAL_COL","NORMAL_COL","PARTITION_COL"]},"filter":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":1,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":2,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{"field":3}}}}},{"value":{"literal":{"string":"0"}}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":1,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":2,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{"field":3}}}}},{"value":{"literal":{"string":"NULL"}}}]}}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":3,"outputType":{"bool":{"nullability":"NULLABILITY_REQUIRED"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{"field":3}}}}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":1,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":2,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{}}}}},{"value":{"literal":{"string":"NULL"}}}]}}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":3,"outputType":{"bool":{"nullability":"NULLABILITY_REQUIRED"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{}}}}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":1,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":2,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{}}}}},{"value":{"literal":{"string":"000000000000000"}}}]}}}]}}}]}}},{"value":{"singularOrList":{"value":{"selection":{"directReference":{"structField":{"field":1}}}},"options":[{"literal":{"string":"1"}},{"literal":{"string":"2"}},{"literal":{"string":"9"}},{"literal":{"string":"10"}},{"literal":{"string":"11"}},{"literal":{"string":"22"}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":1,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":2,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{"field":3}}}}},{"value":{"literal":{"string":""}}}]}}}]}}}]}}},{"value":{"scalarFunction":{"functionReference":1,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"scalarFunction":{"functionReference":2,"outputType":{"bool":{"nullability":"NULLABILITY_NULLABLE"}},"arguments":[{"value":{"selection":{"directReference":{"structField":{}}}}},{"value":{"literal":{"string":""}}}]}}}]}}}]}},"advancedExtension":{"optimization":{"@type":"/google.protobuf.StringValue","value":"isMergeTree=0\n"}}}},"expressions":[{"selection":{"directReference":{"structField":{}}}},{"selection":{"directReference":{"structField":{"field":1}}}},{"selection":{"directReference":{"structField":{"field":2}}}},{"selection":{"directReference":{"structField":{"field":3}}}}]}},"groupings":[{"groupingExpressions":[{"selection":{"directReference":{"structField":{"field":3}}}},{"selection":{"directReference":{"structField":{}}}},{"selection":{"directReference":{"structField":{"field":1}}}},{"selection":{"directReference":{"structField":{"field":2}}}}]}],"advancedExtension":{"optimization":{"@type":"/google.protobuf.StringValue","value":"isStreaming=0\nallowFlush=1\n"}}}},"expressions":[{"scalarFunction":{"functionReference":4,"outputType":{"i32":{"nullability":"NULLABILITY_REQUIRED"}},"arguments":[{"value":{"literal":{"i32":42}}},{"value":{"selection":{"directReference":{"structField":{}}}}},{"value":{"selection":{"directReference":{"structField":{"field":1}}}}},{"value":{"selection":{"directReference":{"structField":{"field":2}}}}},{"value":{"selection":{"directReference":{"structField":{"field":3}}}}}]}},{"selection":{"directReference":{"structField":{}}}},{"selection":{"directReference":{"structField":{"field":1}}}},{"selection":{"directReference":{"structField":{"field":2}}}},{"selection":{"directReference":{"structField":{"field":3}}}}]}},"names":["hash_partition_key#279","id2#61","id#49","p1#51","update_time#55","hash_partition_key#279","id2#61","id#49","p1#51","update_time#55"]}}]}
```

### Gluten version
Gluten-1.2
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start with the supplied YARN ExecutorLostFailure, memory profile, and error plan, focusing on the aggregate and hash stages in the Velox backend. Reproduce the workload if possible and isolate whether aggregation or hash partitioning drives the physical-memory limit; done means the responsible stage and a verified fix or mitigation are identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100