Azure / Azure/azureml-examples

Pipeline support for Command Job & Ray Distribution

Open
#2,929 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

### Describe your suggestion

I cant find a way to run a command job as a repeatable pipeline using Ray Distribution. Ray Distribution works great with a plain command job -- but I want to submit a pipeline and run it regularly via Synapse.

Submitting command jobs through pipeline decorators turns the commands into MPI runs which fail and are incompatible.

### Additional details

_No response_

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating the pipeline decorator and Command Job examples that configure Ray Distribution, then compare their submission paths. Done means a repeatable Synapse pipeline can run a Ray Distribution command job without converting it into an incompatible MPI run.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.