apache / apache/datafusion

Add internal EXPLAIN ANALYZE metric level

Open
#22,155 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Some execution metrics are useful for low-level operator debugging but are too noisy or high-cardinality for the default `EXPLAIN ANALYZE` output. Today the `summary` and `dev` levels do not provide a clean place for these internal development metrics.

### Describe the solution you would like

Add an `internal` `datafusion.explain.analyze_level` that includes `summary`, `dev`, and low-level internal metrics. Operators can use this level for targeted debugging metrics while keeping the default `dev` output concise and avoiding unnecessary metric registration overhead.

### Describe alternatives you have considered

Add operator-specific flags or environment variables for granular metrics. That makes metrics harder to discover and does not compose well across operators.

### Additional context

`RepartitionExec` granular timings are the first intended use case for this level.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how datafusion.explain.analyze_level is parsed and how EXPLAIN ANALYZE selects metrics, then inspect RepartitionExec as the stated first use case. Done means an internal level includes summary, dev, and low-level internal metrics while dev remains concise and avoids unnecessary metric registration overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.