datafusion-contrib / datafusion-contrib/datafusion-distributed

[Epic] Abstract transportation protocol details

Open
#580 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
139
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
35

Description

Decoupling the codebase from transportation details will unlock two things:

  1. People who need a transportation other then gRPC can implement it while keeping the core logic of the project intact
  2. We can have an in-memory implementation that is always chosen for intra-worker comms

While this was helpful for keeping a decoupled codebase, it also unlocks further optimizations:

  • If a worker needs to pull data from another worker, and both happen to be identified with the same URL, it's safe to assume that the worker is trying to talk to itself, so it can fallback to a full in-memory connection.
  • If the coordinator needs to communicate with a worker for feeding it work, and it happens to be co-located with the worker to which it should feed work, it can fallback to in-memory comms without paying network + serialization.
  • If the coordinator decides that one task is more than enough for certain stages, it can colocate them in itself, and essentially execute the plan fully in-memory as if it was single-node.
  • The AQE machinery can be used in a single-node context, relying fully on in-memory comms.

This is a collection of PRs an issues that will need to happen to consider this task done:

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading PR #512 and issues #576, #578, #577, and #579, since this epic identifies them as the work needed for completion. Trace how each item addresses transport coupling and in-memory communication. Done means the listed work is complete and the project supports the transport and colocated in-memory scenarios described here.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, rust
Domain
distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.