AnswerDotAI / AnswerDotAI/nbdev
Custom Directives and Example: Integration with MetaFlow
- Dominant language
- Jupyter Notebook
- Stars
- 5.3k
- Forks
- 513
- Avg merge
- 2d 30m
- Merged PRs (30d)
- 8
Description
Tagging @hamelsmu since we had talked about this earlier.
#### Description
I was working on a custom use of Nbdev, using it to define my DAGs. This is because I usually prototype in a notebook and then copy the important code when writing my DAGs in my IDE anyway.
Here is an example repo, where I prototype training MNIST with fastai on a notebook and attempt to create the DAG Workflow from the notebook: https://github.com/jimmiemunyi/nbdev-metaflow-example. In the end, I was able to achieve the desired result, but with some hiccups here and there. The notebooks are in the `nbs` folder and the generated code is in the `nbdev_metaflow_example/workflows` folder.
I have also highlighted issues I came across in the notebooks but will also highlight them here in this issue
#### Issues I encountered.
- If we can have custom Directives, we can potentially adapt nbdev to particular use cases when developing code. People could even write extensions to nbdev for specific libraries like `nbdev_metaflow_extension`. An example of working with Metaflow :
- we do not require the `__all__` at the top since the WorkFlows are not meant to be imported. If one could write a custom directive instead of using the default `#| default_exp foo`, I would probably customize it to fit the way Metaflow expects the final file to be.
- we need an `__if__ == '__main__'` at the end of every file to call the workflow. I had to manually add it all the time
```python
if __name__ == '__main__':
SimpleMNISTFlow()
```
- Monkey patching but with passing in a decorator. Suppose I want to monkey patch a method into a class, but the method also requires a decorator, how would I achieve this? This is highlighted in the `01_Complex_MNIST_Example.ipynb`
Example:
```python
@conda(libraries={'numpy':'1.18.1'})
@resources(memory=1000)
@step
def start(self):
"""
Parse the MNIST Dataset into Flattened and None Flattened Data artifacts.
"""
import numpy as np
self.data = np.array([2.0, 3.0])
```
The patch functionality comes from fastcore so this is a big if. I am not sure if it's possible or if it would not be practical:
```python
@patch
def _dummy(self:ComplexMNISTFlow, decorators={'resources': 'memory=1000'}):
pass
```
Anyway, big fan of the library and I use it almost all the time I want to develop some python code.
Contributor guide
Assessment
This issue has not been assessed yet.