calkit / calkit/calkit

I want to write my pipeline in Python so I can create stages programmatically

Open
#649 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
60
Forks
18
Avg merge
1d 16h
Merged PRs (30d)
38

Description

For example:

```python
from calkit.models import Pipeline, PythonScriptStage, SBatchStage

def make_nsys_stage(name: str) -> SBatchStage:
return SBatchStage(name=name, ...)

pipeline = Pipeline(
stages=[
make_nsys_stage("baseline"),
make_nsys_stage("mod"),
make_nsys_stage("const"),
PythonScriptStage(
name="summarize",
script="scripts/summarize.py",
inputs=["results/nsys/baseline.sqlite"],
),
]
)
```

Then in `calkit.yaml`:

```yaml
pipeline:
kind: python
path: pipeline.py
```

Calkit will import the script and look for a single Pipeline object in `locals`, which it will then use to compile the pipeline.

Related to #561

We could also simply use a `calkit.py` file and merge it with `calkit.yaml`, overriding any names.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.