Investigate bridging Spark DataFrames to Dask DataFrames
Open
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Once https://github.com/dask/distributed/issues/566 is implemented we could investigate making it easier for users to copy their Spark DataFrame into Dask and perform additional computation. This could benefit users that want to read from traditional hadoop data sources, such as parquet, in a distributed fashion.
Implementation concerns:
- When a spark worker goes away we need to figure out reconstitute the corresponding dataframe. We can do this easily with the long running one, but its an open question for how dask will handle this temporary loss of data.
Contributor guide
Assessment
This issue has not been assessed yet.