datafusion-contrib / datafusion-contrib/datafusion-distributed
add a better display API
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
In #623, we add a new rewrite API prior to displaying:
let plan =
rewrite_distributed_plan_with_dynamic_filters(plan).await?;
// 4. Fold the per-task metrics back into the plan...
let plan =
rewrite_distributed_plan_with_metrics(plan, DistributedMetricsFormat::Aggregated).await?;
// 5. ...and render it.
println!("{}", display_plan_ascii(plan.as_ref(), true));
It's annoying to have to call those two rewrites.
Ideally, if you set with_distributed_dynamic_filter_collection or with_distributed_metrics_collection, then displaying is handled automatically. One downside to this is that those settings are stored in the SessionConfig, so you would have to pass a SessionConfig to display_plan_ascii, which isn't great.
Another way to do it is to add options to explain_analyze or display_plan_ascii
DisplayOptions {
metrics_mode: Aggregated | PerTask
dynamic_filters: bool
}
and have those methods handle the rewrites for you.
I imagine we want to improve the API for both metrics and
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the explain_analyze implementation in src/stage.rs around the linked line and the display_plan_ascii call shown in the issue. Compare the proposed DisplayOptions for metrics_mode and dynamic_filters with the existing rewrite calls. Done means the display or explain API handles the requested rewrites without requiring callers to invoke both manually, with the API design resolved for both metrics and dynamic filters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100