I want to be able to process/query big datasets reproducibly without needing to download to my local machine
Open
- Dominant language
- Python
- Stars
- 60
- Forks
- 18
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 38
Description
## Concepts
1. Users can define a distributed processing environment like Spark, Dask, Trino, etc., and a command could run in that.
2. We could spin up a large machine for them, clone the project onto it, then run a stage or the whole pipeline there, commit, push, and they could pull down the result. This wouldn't be a great interactive workflow though. It i s probably a requirement that interactive tasks be able to be done to build the pipeline.
Contributor guide
Assessment
This issue has not been assessed yet.