Pushing down HashJoinExec build side dynamic filters makes tpch queries slower

### Describe the bug

Seen / discussed in:

https://github.com/apache/datafusion/pull/19639
https://github.com/apache/datafusion/issues/3463

We need to nail down what is slow:
1. Creating the expressions?
2. Evaluating them?

If (2), what's the solution?
- Is our approach to filter pushdown fundamentally flawed because it changes the parallelism or batch sizes at which the filter is applied? Could we do something to make the expression more efficient, like put the min/max evaluation outside of the hashes?
- Is it just because the hashes are being evaluated more times?
- Is it less of a problem on a larger dataset?

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_