datafusion-contrib / datafusion-contrib/datafusion-distributed

Allow workers to reject work for a query (before side effects)

Open
#640 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
139
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
35

Description

Right now, once the coordinator selects a worker for a query task, that worker is committed to the query and any failure in the worker will fail the entire query, across many workers. This commitment happens before any communication with the worker.

For load balancing, rolling restarts, etc - it is valuable for a worker to be able to locally decide to refuse some new work to maintain throughput.

I see two options to achieve this:

  1. The worker returns a yes/no to the coordinator about accepting the work (this can happen as part of establishing the initial gRPC stream), and the coordinator retries on a different worker only on an explicit 'no' which is guaranteed to be sent before any side effects.

  2. After establishing the gRPC stream, the worker waits for an explicit 'go-ahead' from the coordinator before executing anything side-effectfull (e.g., pulling data from lower stages in the plan). This means that timeouts and other ambiguous failures can also be retried on other workers from the coordinator, as long as the coordinator has not sent a 'go-ahead'.

My preference is for option 2, as this provides more ability to work around an unhealthy worker with a cost of ~half a coordinator-worker round-trip latency.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the coordinator's worker-selection flow and the worker gRPC stream through the point where query work can cause side effects. Compare the two proposed handshake designs, then define the retry and failure semantics before identifying the protocol changes and tests needed to show that rejected or ambiguous work can move to another worker safely.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.