Azure / Azure/azure-sdk-for-python
queue settings is ignored when deploying pipeline behind batch endpoint
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
- **Package Name**: azure-ai-ml
- **Package Version**: 1.29.0
- **Operating System**: Linux
- **Python Version**: 3.11.3
**Describe the bug**
When deploying a batch endpoint using a pipeline component, the `queue_settings` (such as `job_tier` and `priority`) configured in the component are not propagated to the batch endpoint deployment. The jobs run with default settings, ignoring the specified `queue_settings`.
**To Reproduce**
Steps to reproduce the behavior:
1. Define a pipeline component with `command` and set `queue_settings` (e.g., `job_tier="Spot"`, `priority="LowPriority"`).
2. Deploy the pipeline as a batch endpoint using `PipelineComponentBatchDeployment`.
3. Observe that the batch endpoint jobs do not use the specified `queue_settings`.
**Minimal code example:**
```python
from azure.ai.ml import MLClient, command, dsl
from azure.ai.ml.entities import CommandComponent, JobResourceConfiguration, Environment
from azure.identity import DefaultAzureCredential
from azure.ai.ml.entities import PipelineComponentBatchDeployment
# ...existing code to authenticate and create MLClient...
env = Environment(
name="debugpipeline",
image="acrindiasocledfc.azurecr.io/gdo/features-engineering-core:features-custom-env",
datastore="workspaceblobstore",
)
hello_world = command(
code="/path/to/hello_world.py",
name="debug_hello_world",
command="python hello_world.py",
environment=env,
resources=JobResourceConfiguration(
instance_type="Standard_D2ds_v5",
instance_count=1,
instance_priority="low",
),
job_tier="Spot",
priority="LowPriority"
)
@dsl.pipeline(name="debug_pipeline")
def debug_pipeline():
hello_world()
return {}
deployment = PipelineComponentBatchDeployment(
name="dep-debug-pipeline",
description="Debug pipeline",
endpoint_name="ep-debug-pipeline",
component=debug_pipeline().component,
settings={"default_compute": "serverless", "continue_on_step_failure": "false"},
tags={"cp-deploiement": "manuel"},
)
ml_client.batch_deployments.begin_create_or_update(deployment)
```
**Expected behavior**
The batch endpoint jobs should respect the `queue_settings` defined in the pipeline component, running with the specified `job_tier` and `priority`. Whereas it works for resources config.
**Additional context**
If this behavior is expected, how can we force the `job_tier` or other `queue_settings` at batch endpoint invocation time?
edit: i fixed py code
Contributor guide
Assessment
This issue has not been assessed yet.