Azure / Azure/MachineLearningNotebooks

ParallelRunStep on Intermediate Partitioned File Dataset Failing

Abierto
#1,648 4 comentarios 1 reacción 0 asignados Ver en GitHub
Lenguaje dominante
Jupyter Notebook
Estrellas
4.4k
Forks
2.6k
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

Hi,

I am attempting to use `ParallelRunStep` for a batch training job. The data that is being ingested for training is a partitioned file dataset, so I am using [this example notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/parallel-run/file-dataset-partition-per-folder.ipynb) as a template.

In my script, the batch training step is preceded by a data pulling step, so the `input` to the training step is of type `OutputDatasetConfig` rather than `Dataset.File.from_files`, which is how it is done in the example notebook. Because of this, the training step fails with the following error:

azureml_common.parallel_run.exception_info.Exception: Run failed. Below is the error detail:
Traceback (most recent call last):
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/azureml/7d68a66e-aa64-4732-a159-22d477774715/driver/simulator.py", line 93, in main
simulator.wait()
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/7d68a66e-aa64-4732-a159-22d477774715/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/masterless_simulator.py", line 122, in wait
ProgressReport().save()
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/7d68a66e-aa64-4732-a159-22d477774715/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/progress_report.py", line 191, in save
task_exporter.save()
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/7d68a66e-aa64-4732-a159-22d477774715/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/task_exporter.py", line 113, in save
self.save_remaining()
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/7d68a66e-aa64-4732-a159-22d477774715/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/task_exporter.py", line 100, in save_remaining
for task in total_tasks:
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/7d68a66e-aa64-4732-a159-22d477774715/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/partition_by_keys_provider.py", line 70, in get_tasks
DatasetHelper().save(dataset)
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/azureml/7d68a66e-aa64-4732-a159-22d477774715/driver/azureml_common/parallel_run/dataset_helper.py", line 29, in save
self.logger.info("Dump dataset {} as preppy files to local directory.".format(dataset.id))
AttributeError: 'str' object has no attribute 'id'

Could this possibly be an issue with `ParallelRunStep` being unable to accept a partitioned `OutputDatasetConfig` as an `input`? Thanks in advance!

---
#### Document Details

⚠ *Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.*

* ID: f69044d5-213e-a764-31dd-24f8368212b7
* Version Independent ID: 23d38b1c-974a-b2fc-332a-70d7500e1751
* Content: [azureml.pipeline.steps.ParallelRunStep class - Azure Machine Learning Python](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.parallelrunstep?view=azure-ml-py)
* Content Source: [AzureML-Docset/stable/docs-ref-autogen/azureml-pipeline-steps/azureml.pipeline.steps.ParallelRunStep.yml](https://github.com/MicrosoftDocs/MachineLearning-Python-pr/blob/live/AzureML-Docset/stable/docs-ref-autogen/azureml-pipeline-steps/azureml.pipeline.steps.ParallelRunStep.yml)
* Service: **machine-learning**
* Sub-service: **core**
* GitHub Login: @DebFro
* Microsoft Alias: **debfro**

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Empieza con el notebook de ejemplo partition-per-folder referenciado y la documentación de ParallelRunStep, y luego compara su entrada Dataset.File.from_files con la entrada OutputDatasetConfig indicada. Reproduce el traceback alrededor de dataset_helper.py y determina si los tipos de entrada documentados cubren este caso; se considera terminado cuando el comportamiento o el uso compatible están claramente establecidos.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
azure, jupyter-notebook, machine-learning, python
Área
machine-learning
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Estancado
Claridad
Necesita aclaración
Aptitud para principiantes
32/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.