datafusion-contrib / datafusion-contrib/datafusion-distributed

add a better display API

Open
#678 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
139
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
35

Description

In #623, we add a new rewrite API prior to displaying:

let plan =
    rewrite_distributed_plan_with_dynamic_filters(plan).await?;

// 4. Fold the per-task metrics back into the plan...
let plan =
    rewrite_distributed_plan_with_metrics(plan, DistributedMetricsFormat::Aggregated).await?;

// 5. ...and render it.
println!("{}", display_plan_ascii(plan.as_ref(), true));

It's annoying to have to call those two rewrites.

Ideally, if you set with_distributed_dynamic_filter_collection or with_distributed_metrics_collection, then displaying is handled automatically. One downside to this is that those settings are stored in the SessionConfig, so you would have to pass a SessionConfig to display_plan_ascii, which isn't great.

Another way to do it is to add options to explain_analyze or display_plan_ascii

DisplayOptions {
    metrics_mode: Aggregated | PerTask
    dynamic_filters: bool
}

and have those methods handle the rewrites for you.

I imagine we want to improve the API for both metrics and

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the explain_analyze implementation in src/stage.rs around the linked line and the display_plan_ascii call shown in the issue. Compare the proposed DisplayOptions for metrics_mode and dynamic_filters with the existing rewrite calls. Done means the display or explain API handles the requested rewrites without requiring callers to invoke both manually, with the API design resolved for both metrics and dynamic filters.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.