Avoid per-file metric registration in `ParquetSource`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
I am benchmarking a query plan with many partitions and a scan over >12,000 Parquet files. Each file opened by the default `ParquetSource` registers 45 metrics into the one `ExecutionPlanMetricsSet` shared by the whole `DataSourceExec`. This results in half a million synchronous mutex acquisitions and metric registrations.
Dial9 schedule/off-CPU profiling of high-file-count scans consistently identified this mutex as the dominant application-level blocking source. The captured stacks ran from Parquet file opening through `MetricBuilder::build` and `ExecutionPlanMetricsSet::register`, where workers entered `futex_wait` while acquiring the shared `parking_lot::Mutex`. Because registration uses a synchronous mutex on Tokio runtime threads, concurrent file opens create a lock convoy: worker threads park, runnable tasks accumulate in the Tokio queue, and CPU remains underutilized.
This per-file detail is not used by normal queries or `EXPLAIN ANALYZE`, which aggregates metrics by name and discards filename labels. The filename-labelled metrics are only exposed through `EXPLAIN ANALYZE VERBOSE` or direct inspection of unaggregated metrics.
### Describe the solution you'd like
Add a metrics cardinality `ConfigOption`:
```text
datafusion.execution.metrics_cardinality = compact | verbose
```
Default to `compact`.
With `compact` cardinality, the `ParquetSource` would:
- Register one metric set per execution partition
- Omit filename labels
- Share the handles across files and replacement readers
With `verbose` cardinality, the `ParquetSource` would preserve per-file filename-labelled metrics.
This changes metric registration growth from `O(files)` to `O(partitions)`.
To maintain existing behavior, `AnalyzeExec` will handle `EXPLAIN ANALYZE VERBOSE` by mutating the child `TaskContext` enabling verbose cardinality (no other `ExecutionPlan` edits its `ConfigOptions`, but it felt appropriate here). Other use cases that directly consume per-file metrics could also explicitly configure it.
### Describe alternatives you've considered
A larger metrics-system redesign could introduce lock-free or sharded registration, typed metric scopes, cardinality limits, or execution-local registries. That may be worthwhile, but it is substantially broader than the Parquet contention problem.
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.