daphne-project / daphne-project/daphne

Distributed object meta data

Open
#409 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
81
Forks
83
PR merge metrics
No merged PRs in 30d

Description

`DistributedWrapper.h` is responsible for:

1. Providing workers with data (pipeline inputs).
2. Broadcasting MLIR code fragment (operations for a given pipeline).
3. Collect results.

During step 1. inputs are distributed across nodes, by splitting rows (or columns/tiles). During this step the object's meta data fields are updated containing information regarding how/where data is placed. This is necessary since during step (2) we read these meta data fields in order to instruct the workers to compute a pipeline.

However, in case an input appears more than two times as an input in the pipeline, we end up needing to update the given object's metadata twice. Also in case we need to distribute an object in two different ways, this furter complicates things. This is issue is to further discuss how we should approach this.

E.g.: `scripts/algorithms/gnmf.daph` the 3rd distributed pipeline has 7 inputs and 3 outputs. `inputs[2]` and `inputs[6]` point to the same matrix, the first input needs to be distributed while the second needs to be broadcasted. This 1. can result in unnecessary communication (we can simply broadcast once) and 2. we need to figure out how meta data should be handled on the same object.

Related issue #367.

Contributor guide

Open the contributing guide

Research direction

Start with DistributedWrapper.h and the third distributed pipeline in scripts/algorithms/gnmf.daph, then trace how repeated inputs and differing distribution modes update metadata. Compare inputs[2] and inputs[6], and read related issue #367 for context. Done should be a documented design that avoids unnecessary communication and defines metadata behavior for repeated objects.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.