apache / apache/dolphinscheduler-sdk-python

[impv] Auto create the whole task workspace to make the task the same workspace when running locally and on the DolphinScheduler platform

Ouverte
#59 0 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
Langage dominant
Python
Étoiles
71
Forks
26
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

For example, If I have the workspace like this: https://github.com/pytorch/examples/tree/main/word_language_model
image

We can submit the task
```python
# [start workflow]
from pydolphinscheduler.core import Workflow
from pydolphinscheduler.core.resource import Resource, LocalResource
from pydolphinscheduler.tasks import Shell

with Workflow(
name="auto_resources_example",
) as workflow:
# [start use_exists_resources]
task_use_resource = Shell(
name="example",
command=f"python main.py --cuda",
auto_resource=True
)
# [end use_exists_resources]

workflow.run()
# [end workflow]
```

Then, all the files will be uploaded to the resource center.
And there is another best way to track the file version, If this is the git repository, we only record the git repository, version, and diff.
Then, we can reproduce the workspace in dolphinscheduler. (Inspired by ClearML)
So, After developing a project locally, we can directly submit it to DolphinScheduler and run it directly. Except for some inconsistent environment variables.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Start with the pydolphinscheduler.core.Workflow and tasks.Shell entry points shown in the example, including workflow.run() and auto_resource=True; the example workspace is word_language_model with main.py. Determine how the resource center currently receives files, then define what is needed for a local workspace or Git repository version and diff to be reproducible on DolphinScheduler, while accounting for inconsistent environment variables.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
git, python
Domaine
backend, devops
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
35/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.