comnik / comnik/declarative-dataflow
Attribute Distribution
- Dominant language
- Rust
- Stars
- 326
- Forks
- 27
- PR merge metrics
- No merged PRs in 30d
Description
Some relations (such as Datomic's `:db/ident`, or relations carrying metadata) are comparatively small but can induce massive skew, especially once reverse indices are involved. This can lead to straggling workers and lots of exchange data.
It might therefore be useful to introduce a new dimension of attribute configuration
```
Distribution := Sharded | Shuffled | Broadcasted
```
that would allow us to configure small, critical relations as `Broadcasted`, while keeping their would-be-skewed join partners entirely local, or shuffled randomly across workers.
The query engine might then need to enforce some new rules for what types of distribution are allowed to go together in a join.
Contributor guide
Assessment
This issue has not been assessed yet.