Azure / Azure/azure-cli

Sweep job support for pipeline component in Azure Machine Learning

Open
#30,929 1 comment 0 reactions 0 assignees View on GitHub
act-codegen-extensibility-squad Auto-Assign customer-reported Machine Learning Service Attention
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

**Related command**
`az ml component create -f XXX.yaml`
`az ml job create -f XXX.yaml`

**Is your feature request related to a problem? Please describe.**
No.

**Describe the solution you'd like**
We are implementing a Retrieval-Augmented Generation (RAG) system using Azure Machine Learning, structured into multiple components such as:

- `process-data-input`
- `retrieve_docs_from_search_index`
- `build_prompt`
- `query_llm`
- `evaluate`

We want to perform hyperparameter tuning across these components. For example, we aim to optimize parameters such as:

- The number of documents retrieved from the search index
- Different prompt templates
- Various `q` values in LLM queries

However, **AzureML Sweep Jobs currently seem to only support `command components`**, making it impossible to tune hyperparameters across components. We tried to group our `command components` in to a `pipeline component`. But sweep job does not start correctly. It would be highly beneficial if Sweep Jobs could also support `pipeline component`, allowing us to bundle our components into a single `pipeline component` and efficiently perform hyperparameter tuning.

**Describe alternatives you've considered**
Our current approach involves creating a standalone `command component` that pulls source code from all other components. However, this:

- **Defeats the purpose** of separating concerns across multiple components
- **Violates the Single Responsibility Principle** in system design
- **Increases maintenance complexity**

**Additional context**
Extending Sweep Job support to `pipeline component` would greatly improve modularity, maintainability, and efficiency in hyperparameter tuning.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.