AnswerDotAI / AnswerDotAI/nbdev

Custom Directives and Example: Integration with MetaFlow

Open
#1,019 0 comments 0 reactions 1 assignee Claimed by @hamelsmu View on GitHub
Dominant language
Jupyter Notebook
Stars
5.3k
Forks
513
Avg merge
2d 30m
Merged PRs (30d)
8

Description

Tagging @hamelsmu since we had talked about this earlier.

#### Description
I was working on a custom use of Nbdev, using it to define my DAGs. This is because I usually prototype in a notebook and then copy the important code when writing my DAGs in my IDE anyway.

Here is an example repo, where I prototype training MNIST with fastai on a notebook and attempt to create the DAG Workflow from the notebook: https://github.com/jimmiemunyi/nbdev-metaflow-example. In the end, I was able to achieve the desired result, but with some hiccups here and there. The notebooks are in the `nbs` folder and the generated code is in the `nbdev_metaflow_example/workflows` folder.

I have also highlighted issues I came across in the notebooks but will also highlight them here in this issue

#### Issues I encountered.

- If we can have custom Directives, we can potentially adapt nbdev to particular use cases when developing code. People could even write extensions to nbdev for specific libraries like `nbdev_metaflow_extension`. An example of working with Metaflow :
- we do not require the `__all__` at the top since the WorkFlows are not meant to be imported. If one could write a custom directive instead of using the default `#| default_exp foo`, I would probably customize it to fit the way Metaflow expects the final file to be.
- we need an `__if__ == '__main__'` at the end of every file to call the workflow. I had to manually add it all the time

```python
if __name__ == '__main__':
SimpleMNISTFlow()
```

- Monkey patching but with passing in a decorator. Suppose I want to monkey patch a method into a class, but the method also requires a decorator, how would I achieve this? This is highlighted in the `01_Complex_MNIST_Example.ipynb`
Example:

```python
@conda(libraries={'numpy':'1.18.1'})
@resources(memory=1000)
@step
def start(self):
"""
Parse the MNIST Dataset into Flattened and None Flattened Data artifacts.
"""
import numpy as np

self.data = np.array([2.0, 3.0])
```
The patch functionality comes from fastcore so this is a big if. I am not sure if it's possible or if it would not be practical:

```python
@patch
def _dummy(self:ComplexMNISTFlow, decorators={'resources': 'memory=1000'}):
pass

```

Anyway, big fan of the library and I use it almost all the time I want to develop some python code.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.