apache / apache/datafusion

Need to add an unique id to MetricsSet or ExecutionPlanMetricsSet

Open
#3,033 7 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)

The is useful for Ballista Scheduler to post merge the metrics of the plan operators from all the Executor task results.
Today, this is a missing part in Ballista. The task results returned from the executors do not report back any of the plan node execution metrics to Scheduler. After adding the report back feature, the metrics need to be correctly merged, need a way to uniquely identify which metrics belong to which plan operator and only the metrics belong to the same plan node can be merged
in the Scheduler side.

@alamb
Please share your thoughts and advice.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start by tracing MetricsSet and ExecutionPlanMetricsSet through Ballista executor task results and the Scheduler. Determine how metrics for plan operators are identified and merged; done means metrics from the same plan node can be uniquely identified and merged correctly in the Scheduler.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.