comnik / comnik/declarative-dataflow

Attribute Distribution

Open
#81 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
326
Forks
27
PR merge metrics
No merged PRs in 30d

Description

Some relations (such as Datomic's `:db/ident`, or relations carrying metadata) are comparatively small but can induce massive skew, especially once reverse indices are involved. This can lead to straggling workers and lots of exchange data.

It might therefore be useful to introduce a new dimension of attribute configuration

```
Distribution := Sharded | Shuffled | Broadcasted
```

that would allow us to configure small, critical relations as `Broadcasted`, while keeping their would-be-skewed join partners entirely local, or shuffled randomly across workers.

The query engine might then need to enforce some new rules for what types of distribution are allowed to go together in a join.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.