Azure / Azure/MachineLearningNotebooks

More details of DatabricksStep outputs

Ouverte
#1,723 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Jupyter Notebook
Étoiles
4.4k
Forks
2.6k
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

[Enter feedback here]
In my use case DatabricksStep generates a couple of files that should be passed to downstream steps using `PipelineData`. I am using python script from a local compute by specifying `python_script_name `. The script also parses arguments and outputs looked similar to examples in document shown below.

`If you specify a DataReference object as input with data_reference_name=input1 and a PipelineData object as output with name=output1, then the inputs and outputs will be passed to the script as parameters. This is how they will look like and you will need to parse the arguments in your script to access the paths of each input and output: "-input1","wasbs://test@storagename.blob.core.windows.net/test","-output1", "wasbs://test@storagename.blob.core.windows.net/b3e26de1-87a4-494d-a20f-1988d22b81a2/output1"`

So I am using following codes to dump a dummy file.
`
with open(args.output_name, 'w') as f:
json.dump({'hello':'world'}, f)
`
`PipelineData` is a file so no need to check the existence. The step didnot have any error and succeeded. But when I check the file from the blob storage there was not any folder generated (including the folder with Run_ID). This method works for `PythonScriptStep`. The difference is that in `PythonScriptStep`, the output argument is a mounted path like `/mnt/azureml/cr/j/******`

I also tested [OutputFileDatasetConfig](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/%5Bhttps://aka.ms/pl-data-output-config%5D(https://aka.ms/pl-data-output-config)?view=azure-ml-py) as it is a recommended way from [doc](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelinedata?view=azure-ml-py). No error, but have the same issue as `PipelineData` file was not generated.

Are there more details of how to use this inputs/outputs from python scripts?

---
#### Document Details

⚠ *Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.*

* ID: 5d4d04b0-78b3-db32-a338-65f0ffc9167e
* Version Independent ID: 06721cca-fc1c-771c-c036-109712a1f7be
* Content: [azureml.pipeline.steps.databricks_step.DatabricksStep class - Azure Machine Learning Python](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.databricksstep?view=azure-ml-py#methods)
* Content Source: [AzureML-Docset/stable/docs-ref-autogen/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.DatabricksStep.yml](https://github.com/MicrosoftDocs/MachineLearning-Python-pr/blob/live/AzureML-Docset/stable/docs-ref-autogen/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.DatabricksStep.yml)
* Service: **machine-learning**
* Sub-service: **core**
* GitHub Login: @DebFro
* Microsoft Alias: **debfro**

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Commencez par la source de documentation de DatabricksStep à l’emplacement AzureML-Docset/stable/docs-ref-autogen/azureml.pipeline.steps.databricks_step.DatabricksStep.yml et examinez les références liées à PipelineData et OutputFileDatasetConfig. Reproduisez le cas décrit d’un script Python sur local-compute et comparez-le au comportement de PythonScriptStep. C’est terminé lorsque la documentation explique clairement les chemins d’entrée et de sortie pris en charge, la persistance des fichiers et toute limitation pertinente.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
azure, python
Domaine
documentation, machine-learning
Type d'issue
Documentation
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
À clarifier
Accessibilité débutants
28/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.