calkit / calkit/calkit

I want to be able to process/query big datasets reproducibly without needing to download to my local machine

Open
#308 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
60
Forks
18
Avg merge
1d 16h
Merged PRs (30d)
38

Description

## Concepts

1. Users can define a distributed processing environment like Spark, Dask, Trino, etc., and a command could run in that.
2. We could spin up a large machine for them, clone the project onto it, then run a stage or the whole pipeline there, commit, push, and they could pull down the result. This wouldn't be a great interactive workflow though. It i s probably a requirement that interactive tasks be able to be done to build the pipeline.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.