apache / apache/dolphinscheduler-sdk-python
[impv] Auto create the whole task workspace to make the task the same workspace when running locally and on the DolphinScheduler platform
- 主要语言
- Python
- 星标
- 71
- 派生
- 26
- PR 合并指标
- 30 天内没有已合并 PR
描述
For example, If I have the workspace like this: https://github.com/pytorch/examples/tree/main/word_language_model

We can submit the task
```python
# [start workflow]
from pydolphinscheduler.core import Workflow
from pydolphinscheduler.core.resource import Resource, LocalResource
from pydolphinscheduler.tasks import Shell
with Workflow(
name="auto_resources_example",
) as workflow:
# [start use_exists_resources]
task_use_resource = Shell(
name="example",
command=f"python main.py --cuda",
auto_resource=True
)
# [end use_exists_resources]
workflow.run()
# [end workflow]
```
Then, all the files will be uploaded to the resource center.
And there is another best way to track the file version, If this is the git repository, we only record the git repository, version, and diff.
Then, we can reproduce the workspace in dolphinscheduler. (Inspired by ClearML)
So, After developing a project locally, we can directly submit it to DolphinScheduler and run it directly. Except for some inconsistent environment variables.
贡献指南
调研方向
从示例中所示的 pydolphinscheduler.core.Workflow 和 tasks.Shell 入口点开始,包括 workflow.run() 和 auto_resource=True;示例工作区是包含 main.py 的 word_language_model。确定 Resource Center 当前如何接收文件,然后定义要使本地工作区或 Git 仓库的版本及其 diff 能够在 DolphinScheduler 上复现所需的内容,同时考虑环境变量不一致的情况。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- git, python
- 领域
- backend, devops
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100