Azure / Azure/MachineLearningNotebooks
More details of DatabricksStep outputs
- Linguagem predominante
- Jupyter Notebook
- Estrelas
- 4.4k
- Forks
- 2.6k
- Métricas de merge de PRs
- Nenhum PR com merge em 30d
Descrição
[Enter feedback here]
In my use case DatabricksStep generates a couple of files that should be passed to downstream steps using `PipelineData`. I am using python script from a local compute by specifying `python_script_name `. The script also parses arguments and outputs looked similar to examples in document shown below.
`If you specify a DataReference object as input with data_reference_name=input1 and a PipelineData object as output with name=output1, then the inputs and outputs will be passed to the script as parameters. This is how they will look like and you will need to parse the arguments in your script to access the paths of each input and output: "-input1","wasbs://test@storagename.blob.core.windows.net/test","-output1", "wasbs://test@storagename.blob.core.windows.net/b3e26de1-87a4-494d-a20f-1988d22b81a2/output1"`
So I am using following codes to dump a dummy file.
`
with open(args.output_name, 'w') as f:
json.dump({'hello':'world'}, f)
`
`PipelineData` is a file so no need to check the existence. The step didnot have any error and succeeded. But when I check the file from the blob storage there was not any folder generated (including the folder with Run_ID). This method works for `PythonScriptStep`. The difference is that in `PythonScriptStep`, the output argument is a mounted path like `/mnt/azureml/cr/j/******`
I also tested [OutputFileDatasetConfig](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/%5Bhttps://aka.ms/pl-data-output-config%5D(https://aka.ms/pl-data-output-config)?view=azure-ml-py) as it is a recommended way from [doc](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelinedata?view=azure-ml-py). No error, but have the same issue as `PipelineData` file was not generated.
Are there more details of how to use this inputs/outputs from python scripts?
---
#### Document Details
⚠ *Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.*
* ID: 5d4d04b0-78b3-db32-a338-65f0ffc9167e
* Version Independent ID: 06721cca-fc1c-771c-c036-109712a1f7be
* Content: [azureml.pipeline.steps.databricks_step.DatabricksStep class - Azure Machine Learning Python](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.databricksstep?view=azure-ml-py#methods)
* Content Source: [AzureML-Docset/stable/docs-ref-autogen/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.DatabricksStep.yml](https://github.com/MicrosoftDocs/MachineLearning-Python-pr/blob/live/AzureML-Docset/stable/docs-ref-autogen/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.DatabricksStep.yml)
* Service: **machine-learning**
* Sub-service: **core**
* GitHub Login: @DebFro
* Microsoft Alias: **debfro**
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Direção de pesquisa
Comece pela fonte da documentação de DatabricksStep em AzureML-Docset/stable/docs-ref-autogen/azureml.pipeline.steps.databricks_step.DatabricksStep.yml e revise as referências vinculadas de PipelineData e OutputFileDatasetConfig. Reproduza o caso descrito de um script Python em local-compute e compare-o com o comportamento de PythonScriptStep. O trabalho estará concluído quando a documentação explicar claramente os caminhos de entrada e saída compatíveis, a persistência de arquivos e qualquer limitação relevante.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- azure, python
- Domínio
- documentation, machine-learning
- Tipo de issue
- Documentação
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Status de atividade
- Estagnada
- Clareza
- Precisa de esclarecimento
- Facilidade para iniciantes
- 28/100