datafusion-contrib / datafusion-contrib/datafusion-distributed
Add a non-blocking snapshot API for available distributed task metrics
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
Problem
The public metrics-rewriting API waits for all worker tasks to report metrics. Complete metrics may be unavailable after cancellation, task failure, timeout, or shutdown, even though some reports have already arrived.
There is currently no public way to consume those available metrics. The internal rewriter also assumes every expected task is present.
Requested API
Expose a non-blocking, point-in-time API for available task metrics.
The API should:
- Include every available report.
- Tolerate missing reports and gaps in task numbers.
- Preserve existing metrics formats.
- Indicate whether the snapshot is complete.
- Leave the existing complete-metrics API unchanged.
No protocol or worker-reporting changes should be required.
Context
Partial metrics are useful for diagnostics, monitoring, accounting, and resource management. Consumers should be able to use available information without waiting indefinitely or depending on internal implementation details.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the public metrics-rewriting API and the internal rewriter, then trace how worker reports are collected and how missing task numbers are handled. The change is done when a non-blocking snapshot includes all available reports, preserves existing formats, indicates completeness, and leaves the existing complete-metrics API unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100