[VL] TPC-DS 30TB regression on 1.2-rc1 vs 1.1 on hash operations

### Backend

VL (Velox)

### Bug description

There is a significant regression on TPC-DS between releases 1.1 and 1.2. For example, q95 went from ~165s to ~315s on TPC-DS 30TB and it looks like hash build/hash probe operations are the ones which have regressed the most (see images)

Other queries with similar regressions caused by hash operations are q23a(240s to 440s), q23b (similar to q23a), q47(85s to 101s)

I was able to reproduce this issue on both spark 3.4 and spark 3.5 - this is not a query plan change issue. I believe the regression happens in Velox but I'm not sure how to isolate it.

### Spark version

Spark-3.4.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_