Support some form of @imports decorator and task_imports that would inherit the container and create the dependency directly
- Dominant language
- Go
- Stars
- 7.5k
- Forks
- 886
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 120
Description
### Motivation: Why do you think this is important?
Currently, the rate of development of both ML DevOps and areas like GenAI mean that they are way too many papers and technologies for a single person to evaluate, so I expect that these two categories will be split into two different types of jobs.
flyte so far is the best platform I evaluated as it hits open source, features hidden enough to not give big companies trade secrets away, AI bots and extensibility. I've just evaluated flyte for 5 days, so I might be missing things so far, so feel free to correct me.
One thing I see is missing in the documentaion and the DSL is: inheritance of container types, imports in tasks and easier way to handl promises, than the ">>" operator in workflows.
Ideally as a DevOps partitioner, I would like set up:
```
@imports # new decorator type
def genai_train(container_image=mypythonenv) #imagespec or config env
import pandas
# just import commands
# instead of repeating the container image, use:
# task_imports. It would: create a dependency, container image and bring in the imports of genai_train()
@task(task_imports=genai_train)
def genai_train(...) -> StructuredDataSet
..etc..
```
By having the imports inside a new decorator type it isolates the environment, allows DevOps easy maintenaince, and frees GenAI partitioners from having to concern themselves with the DevOps environment.
Having the imports isolated within a function has two benefits:
1) You don't have to necessary mimic the local environment when starting a task or waste disk space in a python virtual container with the same imports you will be using remotely. This is a major benefit to those working remotely.
2) You avoid the DRY principle, of having to specifiy the image on each task which is **EXTREMELY** repetitive.
### Goal: What should the final outcome look like, ideally?
DRY principle. Isolate DevOps from researchers.
### Describe alternatives you've considered
Keep using ImageSpec and >> promises.
### Propose: Link/Inline OR Additional context
_No response_
### Are you sure this issue hasn't been raised already?
- [X] Yes
### Have you read the Code of Conduct?
- [X] Yes
Contributor guide
Assessment
This issue has not been assessed yet.