apache / apache/datafusion

Improve metrics in `DataSourceExec` with `Parquet` source

Open
#18,195 6 comments 3 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

When I am working on setting the default metrics in parquet scanner in https://github.com/apache/datafusion/issues/18116, I have several ideas to further improve the metrics accounting in `EXPLAIN ANALYZE` for the parquet scanner.

- [x] Support a new metric value `files_ranges_matched_statistics` https://github.com/apache/datafusion/pull/18297
- [x] Add a new metric value `scan_efficiency_ratio` https://github.com/apache/datafusion/pull/18577
- [ ] Fix `elapsed_compute` baseline metrics not counting issue
- [x] Add a new metric type for the general pruning-related metrics https://github.com/apache/datafusion/pull/18297

### Support a new metric value `files_ranges_matched_statistics`
There is a existing metric `files_ranges_pruned_statistics`
https://github.com/apache/datafusion/blob/155b56e521d75186776a65f1634ee03058899a79/datafusion/datasource-parquet/src/metrics.rs#L44
It would be good also to display how many files ranges are matched to make it more comprehensive, similar to the existing row-group/page level metrics.

### Add a new metric value `scan_efficiency_ratio`
I think it would be helpful to track:
```
scan_efficiency_ratio -- bytes_scanned / total_file_size, as a quick insight for the overall pruning effectiveness
```

### Fix `elapsed_compute` baseline metrics not counting issue
It seems currently the `elapsed_compute` baseline metric is not tracked, you can try any whole file scan on parquet source in `datafusion-cli`, the metric will be unrealistically low:
```
DataFusion CLI v50.2.0
> CREATE EXTERNAL TABLE IF NOT EXISTS lineitem
STORED AS parquet
LOCATION '/Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem';
0 row(s) fetched.
Elapsed 0.049 seconds.

> explain analyze
select * from lineitem;

...elapsed_compute = 14ns...
```

### Add a new metric type for the general pruning-related metrics
There are many levels of pruning inside parquet scanner: file range/row group stat/row group bloom filter/page index, ...
It's currently displayed like ` row_groups_matched_statistics=3, row_groups_pruned_statistics=7`

I think display it as `row_groups_statistics_pruning= 10 total -> 3 matched` looks better, and can make the lengthy existing metrics output more concise.

To do it, we can add a new metric value type, and change its display implementation.

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/datasource-parquet/src/metrics.rs and reproduce the whole-file Parquet scan shown in datafusion-cli with EXPLAIN ANALYZE. Trace how the elapsed_compute baseline metric is accounted for during the Parquet scan. Done means the metric reflects compute time rather than the unrealistically low value reported in the example.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.