Support multi-partitions when compiling DataFusion to `wasm` target
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
This test case will fail (assuming #15595 is merged):
```rust
#[wasm_bindgen_test(unsupported = tokio::test)]
async fn test_multiple_partitions() {
use futures::StreamExt;
let ctx = SessionContext::new();
let dummy_schema = Schema::new(vec![Field::new("a", DataType::Int64, false)]);
let placeholder =
datafusion_physical_plan::placeholder_row::PlaceholderRowExec::new(Arc::new(
dummy_schema,
))
.with_partitions(2);
let task_ctx = ctx.task_ctx();
let plan =
datafusion_physical_plan::coalesce_partitions::CoalescePartitionsExec::new(
Arc::new(placeholder),
);
let mut stream =
datafusion_physical_plan::ExecutionPlan::execute(&plan, 0, task_ctx).unwrap();
let batch = stream.next().await.unwrap().unwrap();
assert_eq!(batch.num_rows(), 1);
assert_eq!(batch.column(0).len(), 1);
}
```
To reproduce, you need to copy the test to `datafusion/wasmtest/src/lib.rs` and run:
```bash
wasm-pack test --headless --chrome
```
It fails with:
```text
console.log div contained:
panicked at /home/hao/coding/datafusion/datafusion/common-runtime/src/join_set.rs:66:20:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
Stack:
Error
at http://127.0.0.1:39955/wasm-bindgen-test:556:21
at logError (http://127.0.0.1:39955/wasm-bindgen-test:15:18)
at imports.wbg.__wbg_new_78093c5bd701d017 (http://127.0.0.1:39955/wasm-bindgen-test:555:66)
```
However, if you change to `with_partitions(1)`, the test will pass.
Related to https://github.com/apache/datafusion/issues/14478 (might be too advanced for a gsoc project though)
#13815 #13715 #13818
### Describe the solution you'd like
It fails because it tries to span tokio tasks in browser, which does not have a tokio runner.
Instead, it should spawn the task to browser's event_loop, which imo is quite non-trivial.
This requires us to have a wasm-specific implementation of [JoinSet](https://github.com/apache/datafusion/blob/main/datafusion/common-runtime/src/join_set.rs), which spawns tasks to browser event loop rather than tokio runtime.
**A deeper implication is that this also moves DataFusion towards a runtime-agnostic design**. I personally believe this is a good thing, but I'm not sure if it is worth the effort.
### Describe alternatives you've considered
If we do nothing, everything still works, just means that DataFusion on wasm can only use a single partition, thus single threaded.
Or we can wait and see if wasi becomes the standard and "DataFusion on wasm" is not `wasm32-unknown-unknown`, but rather `wasm32-wasi`, which should magically fix the problem. However, I expect this to take many years to become a reality.
(cc @alamb, this is the reason [parquet-viewer](https://parquet-viewer.xiangpeng.systems) stays single-threaded)
### Additional context
Reference: https://github.com/cunarist/tokio-with-wasm
Contributor guide
Research direction
Start with datafusion/wasmtest/src/lib.rs and reproduce the multi-partition case using wasm-pack test --headless --chrome. Then read datafusion/common-runtime/src/join_set.rs and the referenced partition execution code to understand the failing runtime assumption. Done means the multi-partition test passes in the wasm browser environment without requiring a Tokio reactor.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, wasm
- Domain
- backend, web-dev
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100