Azure / Azure/MachineLearningNotebooks

More details of DatabricksStep outputs

Open
#1,723 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
4.4k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

[Enter feedback here]
In my use case DatabricksStep generates a couple of files that should be passed to downstream steps using `PipelineData`. I am using python script from a local compute by specifying `python_script_name `. The script also parses arguments and outputs looked similar to examples in document shown below.

`If you specify a DataReference object as input with data_reference_name=input1 and a PipelineData object as output with name=output1, then the inputs and outputs will be passed to the script as parameters. This is how they will look like and you will need to parse the arguments in your script to access the paths of each input and output: "-input1","wasbs://test@storagename.blob.core.windows.net/test","-output1", "wasbs://test@storagename.blob.core.windows.net/b3e26de1-87a4-494d-a20f-1988d22b81a2/output1"`

So I am using following codes to dump a dummy file.
`
with open(args.output_name, 'w') as f:
json.dump({'hello':'world'}, f)
`
`PipelineData` is a file so no need to check the existence. The step didnot have any error and succeeded. But when I check the file from the blob storage there was not any folder generated (including the folder with Run_ID). This method works for `PythonScriptStep`. The difference is that in `PythonScriptStep`, the output argument is a mounted path like `/mnt/azureml/cr/j/******`

I also tested [OutputFileDatasetConfig](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/%5Bhttps://aka.ms/pl-data-output-config%5D(https://aka.ms/pl-data-output-config)?view=azure-ml-py) as it is a recommended way from [doc](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelinedata?view=azure-ml-py). No error, but have the same issue as `PipelineData` file was not generated.

Are there more details of how to use this inputs/outputs from python scripts?

---
#### Document Details

⚠ *Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.*

* ID: 5d4d04b0-78b3-db32-a338-65f0ffc9167e
* Version Independent ID: 06721cca-fc1c-771c-c036-109712a1f7be
* Content: [azureml.pipeline.steps.databricks_step.DatabricksStep class - Azure Machine Learning Python](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.databricksstep?view=azure-ml-py#methods)
* Content Source: [AzureML-Docset/stable/docs-ref-autogen/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.DatabricksStep.yml](https://github.com/MicrosoftDocs/MachineLearning-Python-pr/blob/live/AzureML-Docset/stable/docs-ref-autogen/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.DatabricksStep.yml)
* Service: **machine-learning**
* Sub-service: **core**
* GitHub Login: @DebFro
* Microsoft Alias: **debfro**

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.