dask / dask/distributed

dask in pyscript

Open
#6,257 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

For those that didn't hear the news, [pyscript](https://pyscript.net/) is a new project from Anaconda, to run a complete CPython runtime in the browser via wasm. I will post the pycon keynote when it becomes available. There are already impressive demos interacting with existing pydata tools (numpy, pandas) as well as hooks into JS display tools (d3...) *without any server at all*.

One thing pyscript does not have is a good data story, because the browser environment doesn't support (TCP) "sockets" , only HTTP(s) as provided by the browser runtime, and limited by CORS. That means that fsspec has a real hard time doing anything useful.

On the other hand, dask is able to talk to a distributed cluster over websockets - this already works today. My question is, is there any interest in pushing dask forward as an in-browser, async-mode client? Interestingly, this would enable full fsspec operation by doing all operations via a worker. From the point of view of Coiled (or others that run a remote scheduler), it would get around the tricky/jhub issue of where to run the python kernel.

Progress would require a complex build chain to wasm-ify dask's dependencies, but I think that so long as we steer away from explicit socket-level stuff by relying on websockets, it should be doable. (noting that the current python websocket stack relies on python's builtin httplib/sockets, so would need rewriting to use the browser's internal JS-facing interface).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.