Azure / Azure/azureml-examples

Using PipelineParameter to set the output file path in OutputFileDatasetConfig for each new run against a PublishedPipeline

Open
#1,091 1 comment 1 reaction 0 assignees View on GitHub
example request pipeline
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

## What example? Describe it

**How can we dynamically change the output file path of `OutputFileDatasetConfig` for a `PublishedPipeline`, since we cannot set variables at the time of publishing a definition. E.g. This would not work `OutputFileDatasetConfig(name="processed_data", destination=(datastore, f"mypath/{today}/{output-dataset-name}")).as_upload()` because there's no way to update the parameterized file path when you submit a run against that PublishedPipeline later on.**

`PipelineParameters`, you say?! Well, when using a PipelineParameter to try to set the output path at runtime, [we get something that looks like this issue.](https://github.com/Azure/azure-sdk-for-python/issues/23744)

So, this does not work:
```
# Create dataset output path from pipeline param so it can be changed at runtime
data_output_path_pipeline_param = PipelineParameter(
name="data_output_path",
default_value='default_value'
)

# Create output dataset
output_data = OutputFileDatasetConfig(
name="dataset_output",
destination=( Datastore(workspace, name='datastore_name'), data_output_path_pipeline_param)
).as_upload(overwrite=True)

### ignoring inputs here for brevity ###

### Create pipeline step ###

# Pass input dataset into step1 and upload output to data_output_path_pipeline_param destination
step1 = PythonScriptStep(
script_name="script.py", # doesn't matter what this does
source_directory="src/",
name="Step 1",
arguments=["--dataset-name", input_dataset_name_pipeline_param],
inputs=[tabular_ds_consumption],
outputs=[output_data]
)

### ignoring publishing here for brevity ###

# Passing in the data output path we want to use for the run
experiment.submit(
published_pipeline, # use the pipeline we defined and published earlier
pipeline_parameters={"input_dataset_name": dataset_name,
"data_output_path": f"base_data_pull/{dataset_name}/{today}/{dataset_name}.parquet"
}
)
```
because the pipeline just "uploads" the OutputFileDataset to a bogus "PipelineParameter_Name:data_output_path_Default:base_data_pull/{dataset_name}/{today}/{dataset_name}.parquet" on the datastore.

## TLDR

How should we pass a PipelineParameter into a Pipeline so that after it's published and we want to submit a run for the PublishPipeline, we can pass exactly where we want the output data to land on a datastore (ADLS or Blob) for that run.

If this isn't the right way to use PipelineParameter, what _is_ in order to get the intended behavior?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the published-run example using OutputFileDatasetConfig, PipelineParameter, PublishedPipeline, and experiment.submit. Investigate how the parameter is serialized into the datastore path; done means a submitted run can place its output at the exact runtime path rather than a literal PipelineParameter path.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cloud, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.