datafusion-contrib / datafusion-contrib/datafusion-distributed
[Epic] Abstract transportation protocol details
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
Decoupling the codebase from transportation details will unlock two things:
- People who need a transportation other then gRPC can implement it while keeping the core logic of the project intact
- We can have an in-memory implementation that is always chosen for intra-worker comms
While this was helpful for keeping a decoupled codebase, it also unlocks further optimizations:
- If a worker needs to pull data from another worker, and both happen to be identified with the same URL, it's safe to assume that the worker is trying to talk to itself, so it can fallback to a full in-memory connection.
- If the coordinator needs to communicate with a worker for feeding it work, and it happens to be co-located with the worker to which it should feed work, it can fallback to in-memory comms without paying network + serialization.
- If the coordinator decides that one task is more than enough for certain stages, it can colocate them in itself, and essentially execute the plan fully in-memory as if it was single-node.
- The AQE machinery can be used in a single-node context, relying fully on in-memory comms.
This is a collection of PRs an issues that will need to happen to consider this task done:
- https://github.com/datafusion-contrib/datafusion-distributed/pull/512
- https://github.com/datafusion-contrib/datafusion-distributed/issues/576
- https://github.com/datafusion-contrib/datafusion-distributed/issues/578
- https://github.com/datafusion-contrib/datafusion-distributed/issues/577
- https://github.com/datafusion-contrib/datafusion-distributed/issues/579
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading PR #512 and issues #576, #578, #577, and #579, since this epic identifies them as the work needed for completion. Trace how each item addresses transport coupling and in-memory communication. Done means the listed work is complete and the project supports the transport and colocated in-memory scenarios described here.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, rust
- Domain
- distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100