TimelyDataflow / TimelyDataflow/differential-dataflow

Suggestion: WebAssembly support

Open
#479 22 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

Hi,
I was interested in seeing if I could run differential-dataflow on the web by compiling to WebAssembly using wasm-pack and it worked out really well! However it required some hacks changes to differential-dataflow and timely-dataflow, specifically:

  • For both packages, replacing std::time::Instant with web_time::Instant - crate: web-time This is because there is no "current time" implemented for the wasm32-unknown-unknown compilation target - so we have to hook it up to the browser's Performance.now() function. In all other contexts this uses the regular std::time::Instant under the hood so shouldn't make any difference.
  • Differential dataflow has a 63 bit shift (1 << 63) which needs to be (1 << 31) in 32-bit WebAssembly.

The changes required are:

I was wondering if you would be interested in incorporating this or would prefer to keep this kind of change as a fork?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the linked timely-dataflow and differential-dataflow compare branches, then check the wasm-pack approach and the wasm32-unknown-unknown constraints described in the issue. Done means reaching a maintainer decision and, if accepted, incorporating equivalent WebAssembly support in both packages while preserving non-WebAssembly behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, wasm
Domain
build-system, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.