datafusion-contrib / datafusion-contrib/datafusion-distributed
Allow workers to reject work for a query (before side effects)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
Right now, once the coordinator selects a worker for a query task, that worker is committed to the query and any failure in the worker will fail the entire query, across many workers. This commitment happens before any communication with the worker.
For load balancing, rolling restarts, etc - it is valuable for a worker to be able to locally decide to refuse some new work to maintain throughput.
I see two options to achieve this:
-
The worker returns a yes/no to the coordinator about accepting the work (this can happen as part of establishing the initial gRPC stream), and the coordinator retries on a different worker only on an explicit 'no' which is guaranteed to be sent before any side effects.
-
After establishing the gRPC stream, the worker waits for an explicit 'go-ahead' from the coordinator before executing anything side-effectfull (e.g., pulling data from lower stages in the plan). This means that timeouts and other ambiguous failures can also be retried on other workers from the coordinator, as long as the coordinator has not sent a 'go-ahead'.
My preference is for option 2, as this provides more ability to work around an unhealthy worker with a cost of ~half a coordinator-worker round-trip latency.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the coordinator's worker-selection flow and the worker gRPC stream through the point where query work can cause side effects. Compare the two proposed handshake designs, then define the retry and failure semantics before identifying the protocol changes and tests needed to show that rejected or ambiguous work can move to another worker safely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100