Azure / Azure/azureml-examples

Provide example of best practice for passing objects (e.g. azureml.core.Dataset) between custom components in pipelines

Open
#1,040 0 comments 0 reactions 0 assignees View on GitHub
example request pipeline
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

## What example? Describe it

In the AMLS Designer it appears that components can have a Dataset output type. What's the equivalent to be able to pass a Dataset object from the output of a custom Component A to the input of custom Component B? The only examples I see about dealing with [inputs/outputs of custom components is with writing out data to paths and using the path as an input to the next component](https://github.com/Azure/azureml-examples/tree/main/cli/jobs/pipelines-with-components/basics/2b_component_with_input_output).

I guess this is simply a limitation of custom components vs components in Designer? That custom components by nature are registered separately from the pipeline run they are composed within, so there's no way to configure passing objects between them, since their scripts are registered separately, other than writing out a pickle file and picking it up in the next component via the output path of it's component? Since Designer is ran from the AMLS workspace runtime, this allows for Dataset objects to be passed between?

So is the best practice for passing Dataset objects (and other objects) between AML Custom Components within an AML Pipeline via setting an output path for a pickle file?

![example of output as Dataset object in Designer](https://user-images.githubusercontent.com/6527200/158493467-d54a2014-91dc-4da6-849e-6c8a771b0d97.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.