apache / apache/datafusion

Support multi-partitions when compiling DataFusion to `wasm` target

Open
#15,599 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

This test case will fail (assuming #15595 is merged):

```rust
#[wasm_bindgen_test(unsupported = tokio::test)]
async fn test_multiple_partitions() {
use futures::StreamExt;
let ctx = SessionContext::new();
let dummy_schema = Schema::new(vec![Field::new("a", DataType::Int64, false)]);
let placeholder =
datafusion_physical_plan::placeholder_row::PlaceholderRowExec::new(Arc::new(
dummy_schema,
))
.with_partitions(2);

let task_ctx = ctx.task_ctx();
let plan =
datafusion_physical_plan::coalesce_partitions::CoalescePartitionsExec::new(
Arc::new(placeholder),
);
let mut stream =
datafusion_physical_plan::ExecutionPlan::execute(&plan, 0, task_ctx).unwrap();
let batch = stream.next().await.unwrap().unwrap();
assert_eq!(batch.num_rows(), 1);
assert_eq!(batch.column(0).len(), 1);
}
```

To reproduce, you need to copy the test to `datafusion/wasmtest/src/lib.rs` and run:
```bash
wasm-pack test --headless --chrome
```

It fails with:

```text
console.log div contained:
panicked at /home/hao/coding/datafusion/datafusion/common-runtime/src/join_set.rs:66:20:
there is no reactor running, must be called from the context of a Tokio 1.x runtime

Stack:

Error
at http://127.0.0.1:39955/wasm-bindgen-test:556:21
at logError (http://127.0.0.1:39955/wasm-bindgen-test:15:18)
at imports.wbg.__wbg_new_78093c5bd701d017 (http://127.0.0.1:39955/wasm-bindgen-test:555:66)
```

However, if you change to `with_partitions(1)`, the test will pass.

Related to https://github.com/apache/datafusion/issues/14478 (might be too advanced for a gsoc project though)
#13815 #13715 #13818

### Describe the solution you'd like

It fails because it tries to span tokio tasks in browser, which does not have a tokio runner.

Instead, it should spawn the task to browser's event_loop, which imo is quite non-trivial.

This requires us to have a wasm-specific implementation of [JoinSet](https://github.com/apache/datafusion/blob/main/datafusion/common-runtime/src/join_set.rs), which spawns tasks to browser event loop rather than tokio runtime.

**A deeper implication is that this also moves DataFusion towards a runtime-agnostic design**. I personally believe this is a good thing, but I'm not sure if it is worth the effort.

### Describe alternatives you've considered

If we do nothing, everything still works, just means that DataFusion on wasm can only use a single partition, thus single threaded.

Or we can wait and see if wasi becomes the standard and "DataFusion on wasm" is not `wasm32-unknown-unknown`, but rather `wasm32-wasi`, which should magically fix the problem. However, I expect this to take many years to become a reality.

(cc @alamb, this is the reason [parquet-viewer](https://parquet-viewer.xiangpeng.systems) stays single-threaded)

### Additional context

Reference: https://github.com/cunarist/tokio-with-wasm

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/wasmtest/src/lib.rs and reproduce the multi-partition case using wasm-pack test --headless --chrome. Then read datafusion/common-runtime/src/join_set.rs and the referenced partition execution code to understand the failing runtime assumption. Done means the multi-partition test passes in the wasm browser environment without requiring a Tokio reactor.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, wasm
Domain
backend, web-dev
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.