Azure / Azure/MachineLearningNotebooks
More details of DatabricksStep outputs
- 主要语言
- Jupyter Notebook
- 星标
- 4.4k
- 派生
- 2.6k
- PR 合并指标
- 30 天内没有已合并 PR
描述
[Enter feedback here]
In my use case DatabricksStep generates a couple of files that should be passed to downstream steps using `PipelineData`. I am using python script from a local compute by specifying `python_script_name `. The script also parses arguments and outputs looked similar to examples in document shown below.
`If you specify a DataReference object as input with data_reference_name=input1 and a PipelineData object as output with name=output1, then the inputs and outputs will be passed to the script as parameters. This is how they will look like and you will need to parse the arguments in your script to access the paths of each input and output: "-input1","wasbs://test@storagename.blob.core.windows.net/test","-output1", "wasbs://test@storagename.blob.core.windows.net/b3e26de1-87a4-494d-a20f-1988d22b81a2/output1"`
So I am using following codes to dump a dummy file.
`
with open(args.output_name, 'w') as f:
json.dump({'hello':'world'}, f)
`
`PipelineData` is a file so no need to check the existence. The step didnot have any error and succeeded. But when I check the file from the blob storage there was not any folder generated (including the folder with Run_ID). This method works for `PythonScriptStep`. The difference is that in `PythonScriptStep`, the output argument is a mounted path like `/mnt/azureml/cr/j/******`
I also tested [OutputFileDatasetConfig](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/%5Bhttps://aka.ms/pl-data-output-config%5D(https://aka.ms/pl-data-output-config)?view=azure-ml-py) as it is a recommended way from [doc](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelinedata?view=azure-ml-py). No error, but have the same issue as `PipelineData` file was not generated.
Are there more details of how to use this inputs/outputs from python scripts?
---
#### Document Details
⚠ *Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.*
* ID: 5d4d04b0-78b3-db32-a338-65f0ffc9167e
* Version Independent ID: 06721cca-fc1c-771c-c036-109712a1f7be
* Content: [azureml.pipeline.steps.databricks_step.DatabricksStep class - Azure Machine Learning Python](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.databricksstep?view=azure-ml-py#methods)
* Content Source: [AzureML-Docset/stable/docs-ref-autogen/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.DatabricksStep.yml](https://github.com/MicrosoftDocs/MachineLearning-Python-pr/blob/live/AzureML-Docset/stable/docs-ref-autogen/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.DatabricksStep.yml)
* Service: **machine-learning**
* Sub-service: **core**
* GitHub Login: @DebFro
* Microsoft Alias: **debfro**
贡献指南
这个仓库没有索引到贡献指南
评估
这个 Issue 还没有评估数据。