[Parquet] predicate cache over reports "cache read" metrics in some cases
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Describe the bug**
- While working on https://github.com/apache/datafusion/pull/17275
I am trying to show that when we disable the cache by setting the max size to zero, the cache doesn't pull get used. To do this I was using the [`records_read_from_cache`](https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/metrics/enum.ArrowReaderMetrics.html#method.records_read_from_cache) metric
To my surprise, it reported rows being read from the cache even when the cache was disabled
I found that the metric reports rows that were read from the "local" cache in addition to the actual global cache:
Specifically, in this code:
https://github.com/apache/arrow-rs/blob/main/parquet/src/arrow/array_reader/cached_array_reader.rs#L202-L235
**To Reproduce**
I set the [`max_predicate_cache_size`](https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.ArrowReaderBuilder.html#method.with_max_predicate_cache_size) size to zero (disable the cache), and the looked at this metric:
https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/metrics/enum.ArrowReaderMetrics.html#method.records_read_from_cache
**Expected behavior**
I expect only rows read from the "global cache" (that may consume memory) to be included in the metrics
**Additional context**
Contributor guide
Research direction
Start in parquet/src/arrow/array_reader/cached_array_reader.rs at the referenced lines, then review ArrowReaderMetrics::records_read_from_cache and ArrowReaderBuilder::with_max_predicate_cache_size. Reproduce with the predicate cache size set to zero and verify that the metric counts only rows read from the global cache, not the local cache.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100