apache / apache/datafusion

Avoid per-file metric registration in `ParquetSource`

Open
#23,644 6 comments 0 reactions 1 assignee Claimed by @mitchsw View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

I am benchmarking a query plan with many partitions and a scan over >12,000 Parquet files. Each file opened by the default `ParquetSource` registers 45 metrics into the one `ExecutionPlanMetricsSet` shared by the whole `DataSourceExec`. This results in half a million synchronous mutex acquisitions and metric registrations.

Dial9 schedule/off-CPU profiling of high-file-count scans consistently identified this mutex as the dominant application-level blocking source. The captured stacks ran from Parquet file opening through `MetricBuilder::build` and `ExecutionPlanMetricsSet::register`, where workers entered `futex_wait` while acquiring the shared `parking_lot::Mutex`. Because registration uses a synchronous mutex on Tokio runtime threads, concurrent file opens create a lock convoy: worker threads park, runnable tasks accumulate in the Tokio queue, and CPU remains underutilized.

This per-file detail is not used by normal queries or `EXPLAIN ANALYZE`, which aggregates metrics by name and discards filename labels. The filename-labelled metrics are only exposed through `EXPLAIN ANALYZE VERBOSE` or direct inspection of unaggregated metrics.

### Describe the solution you'd like

Add a metrics cardinality `ConfigOption`:

```text
datafusion.execution.metrics_cardinality = compact | verbose
```

Default to `compact`.

With `compact` cardinality, the `ParquetSource` would:

- Register one metric set per execution partition
- Omit filename labels
- Share the handles across files and replacement readers

With `verbose` cardinality, the `ParquetSource` would preserve per-file filename-labelled metrics.

This changes metric registration growth from `O(files)` to `O(partitions)`.

To maintain existing behavior, `AnalyzeExec` will handle `EXPLAIN ANALYZE VERBOSE` by mutating the child `TaskContext` enabling verbose cardinality (no other `ExecutionPlan` edits its `ConfigOptions`, but it felt appropriate here). Other use cases that directly consume per-file metrics could also explicitly configure it.

### Describe alternatives you've considered

A larger metrics-system redesign could introduce lock-free or sharded registration, typed metric scopes, cardinality limits, or execution-local registries. That may be worthwhile, but it is substantially broader than the Parquet contention problem.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.