[Performance] High cache latency for high selectivity queries because of filter pushdown

When we have a large parquet file, let's say 40M rows. the following query regresses from cache:

This is the normal query from datafusion:

Executing query: SELECT SUM(target_status_code) FROM parquet_table where ___row_id>11534336 AND ___row_id<42735041 AND target_status_code == 200
+---------------------------------------+
| sum(parquet_table.target_status_code) |
+---------------------------------------+
| 4369091600                            |
+---------------------------------------+
Query completed in: 101.976542ms

While in cache it's almost 2x:

SELECT SUM(target_status_code) FROM parquet_table where ___row_id>11534336 AND ___row_id<42735041 AND target_status_code == 200
Executing query: SELECT SUM(target_status_code) FROM parquet_table where ___row_id>11534336 AND ___row_id<42735041 AND target_status_code == 200
+---------------------------------------+
| sum(parquet_table.target_status_code) |
+---------------------------------------+
| 4369091600                            |
+---------------------------------------+
Query completed in: 179.913458ms

The schema of the file is like this:

Row group 6:  count: 831921  40.41 B records  start: 266996175  total(compressed): 32.062 MB total(uncompressed):67.705 MB
--------------------------------------------------------------------------------
                          type      encodings count     avg size   nulls   min / max
backend_ip                BINARY    Z BB_     831921    3.05 B     0       "0x312E302E3139302E323339" / "0x39392E39392E33372E323238"
backend_port              INT32     Z BB_     831921    2.04 B     0       "24932" / "2147482866"
backend_processing_time   FLOAT     Z BB_     831921    1.97 B     0       "0.378" / "99998.555"
backend_status_code       INT32     Z BB_     831921    0.24 B     0       "200" / "504"
client_ip                 BINARY    Z BB_     831921    3.05 B     0       "0x312E302E3130302E313638" / "0x39392E39392E3230372E313136"
client_port               INT32     Z BB_     831921    1.44 B     0       "80" / "65534"
connection_time           FLOAT     Z BB_     831921    1.98 B     0       "1.547" / "99998.664"
destination_ip            BINARY    Z BB_     831921    3.06 B     0       "0x312E302E3134362E33" / "0x39392E39392E36392E323532"
destination_port          INT32     Z BB_     831921    1.44 B     0       "80" / "65535"
elb_status_code           INT32     Z BB_     831921    0.24 B     0       "200" / "504"
http_port                 INT32     Z BB_     831921    1.44 B     0       "80" / "65535"
http_version              BINARY    Z BB_     831921    0.18 B     0       "0x485454502F312E30" / "0x485454502F322E30"
matched_rule_priority     INT32     Z BB_     831921    1.43 B     0       "0" / "50000"
received_bytes            INT32     Z BB_     831921    2.04 B     0       "1120" / "999996598"
request_creation_time     INT64     Z BB_     831921    0.13 B     0       "1724889600000" / "1724975998000"
request_processing_time   FLOAT     Z BB_     831921    1.98 B     0       "1.789" / "99998.5"
response_processing_time  FLOAT     Z BB_     831921    1.97 B     0       "0.138" / "99998.25"
sent_bytes                INT32     Z BB_     831921    2.03 B     0       "8237" / "999984143"
target_ip                 BINARY    Z BB_     831921    3.05 B     0       "0x312E302E36392E323235" / "0x39392E39392E37362E323034"
target_port               INT32     Z BB_     831921    1.44 B     0       "80" / "65535"
target_processing_time    FLOAT     Z BB_     831921    1.97 B     0       "2.462" / "99998.97"
target_status_code        INT32     Z BB_     831921    0.29 B     0       "200" / "504"
timestamp                 INT64     Z BB_     831921    0.13 B     0       "1724889600000" / "1724975998000"
___row_id                 INT32     Z BB_     831921    1.48 B     0       "6291456" / "7123376"
_id                       BINARY    Z BB_     831921    2.32 B     0       "0x00BA19629601A0246DF2455..." / "0xFDFFD41A629601A0246DF24.

I tried to understand it from the point of view of what's different and got to know we are doing filter pushdown which is good when we have low selectivity but it regresses things when we do for queries which has multiple filters and has a high selectivity as well.

Start by reproducing the provided 40M-row Parquet query and compare the uncached and cached timings. Investigate the cache filter-pushdown path for the multiple-filter, high-selectivity case described here. Done means identifying and addressing the regression, with benchmark results showing the cached query no longer incurs the reported latency increase.