dask / dask/distributed

Investigate bridging Spark DataFrames to Dask DataFrames

Open
#567 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Once https://github.com/dask/distributed/issues/566 is implemented we could investigate making it easier for users to copy their Spark DataFrame into Dask and perform additional computation. This could benefit users that want to read from traditional hadoop data sources, such as parquet, in a distributed fashion.

Implementation concerns:
- When a spark worker goes away we need to figure out reconstitute the corresponding dataframe. We can do this easily with the long running one, but its an open question for how dask will handle this temporary loss of data.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.