apache / apache/arrow-rs

[Parquet] predicate cache over reports "cache read" metrics in some cases

Open
#8,307 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Describe the bug**
- While working on https://github.com/apache/datafusion/pull/17275

I am trying to show that when we disable the cache by setting the max size to zero, the cache doesn't pull get used. To do this I was using the [`records_read_from_cache`](https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/metrics/enum.ArrowReaderMetrics.html#method.records_read_from_cache) metric

To my surprise, it reported rows being read from the cache even when the cache was disabled

I found that the metric reports rows that were read from the "local" cache in addition to the actual global cache:

Specifically, in this code:
https://github.com/apache/arrow-rs/blob/main/parquet/src/arrow/array_reader/cached_array_reader.rs#L202-L235

Image

**To Reproduce**
I set the [`max_predicate_cache_size`](https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.ArrowReaderBuilder.html#method.with_max_predicate_cache_size) size to zero (disable the cache), and the looked at this metric:

https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/metrics/enum.ArrowReaderMetrics.html#method.records_read_from_cache

**Expected behavior**
I expect only rows read from the "global cache" (that may consume memory) to be included in the metrics

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start in parquet/src/arrow/array_reader/cached_array_reader.rs at the referenced lines, then review ArrowReaderMetrics::records_read_from_cache and ArrowReaderBuilder::with_max_predicate_cache_size. Reproduce with the predicate cache size set to zero and verify that the metric counts only rows read from the global cache, not the local cache.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.