datafusion-contrib / datafusion-contrib/datafusion-distributed

Add a non-blocking snapshot API for available distributed task metrics

Open
#685 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
139
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
35

Description

Problem

The public metrics-rewriting API waits for all worker tasks to report metrics. Complete metrics may be unavailable after cancellation, task failure, timeout, or shutdown, even though some reports have already arrived.

There is currently no public way to consume those available metrics. The internal rewriter also assumes every expected task is present.

Requested API

Expose a non-blocking, point-in-time API for available task metrics.

The API should:

  • Include every available report.
  • Tolerate missing reports and gaps in task numbers.
  • Preserve existing metrics formats.
  • Indicate whether the snapshot is complete.
  • Leave the existing complete-metrics API unchanged.

No protocol or worker-reporting changes should be required.

Context

Partial metrics are useful for diagnostics, monitoring, accounting, and resource management. Consumers should be able to use available information without waiting indefinitely or depending on internal implementation details.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the public metrics-rewriting API and the internal rewriter, then trace how worker reports are collected and how missing task numbers are handled. The change is done when a non-blocking snapshot includes all available reports, preserves existing formats, indicates completeness, and leaves the existing complete-metrics API unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.