Azure / Azure/MachineLearningNotebooks

ParallelRunStep on Intermediate Partitioned File Dataset Failing

Aperta
#1,648 4 commenti 1 reazione 0 assegnatari Vedi su GitHub
Lingua principale
Jupyter Notebook
Stelle
4.4k
Fork
2.6k
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Hi,

I am attempting to use `ParallelRunStep` for a batch training job. The data that is being ingested for training is a partitioned file dataset, so I am using [this example notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/parallel-run/file-dataset-partition-per-folder.ipynb) as a template.

In my script, the batch training step is preceded by a data pulling step, so the `input` to the training step is of type `OutputDatasetConfig` rather than `Dataset.File.from_files`, which is how it is done in the example notebook. Because of this, the training step fails with the following error:

azureml_common.parallel_run.exception_info.Exception: Run failed. Below is the error detail:
Traceback (most recent call last):
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/azureml/7d68a66e-aa64-4732-a159-22d477774715/driver/simulator.py", line 93, in main
simulator.wait()
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/7d68a66e-aa64-4732-a159-22d477774715/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/masterless_simulator.py", line 122, in wait
ProgressReport().save()
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/7d68a66e-aa64-4732-a159-22d477774715/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/progress_report.py", line 191, in save
task_exporter.save()
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/7d68a66e-aa64-4732-a159-22d477774715/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/task_exporter.py", line 113, in save
self.save_remaining()
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/7d68a66e-aa64-4732-a159-22d477774715/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/task_exporter.py", line 100, in save_remaining
for task in total_tasks:
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/7d68a66e-aa64-4732-a159-22d477774715/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/partition_by_keys_provider.py", line 70, in get_tasks
DatasetHelper().save(dataset)
File "/mnt/batch/tasks/shared/LS_root/jobs/sdpesp-dev-cvx/azureml/7d68a66e-aa64-4732-a159-22d477774715/wd/azureml/7d68a66e-aa64-4732-a159-22d477774715/driver/azureml_common/parallel_run/dataset_helper.py", line 29, in save
self.logger.info("Dump dataset {} as preppy files to local directory.".format(dataset.id))
AttributeError: 'str' object has no attribute 'id'

Could this possibly be an issue with `ParallelRunStep` being unable to accept a partitioned `OutputDatasetConfig` as an `input`? Thanks in advance!

---
#### Document Details

⚠ *Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.*

* ID: f69044d5-213e-a764-31dd-24f8368212b7
* Version Independent ID: 23d38b1c-974a-b2fc-332a-70d7500e1751
* Content: [azureml.pipeline.steps.ParallelRunStep class - Azure Machine Learning Python](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.parallelrunstep?view=azure-ml-py)
* Content Source: [AzureML-Docset/stable/docs-ref-autogen/azureml-pipeline-steps/azureml.pipeline.steps.ParallelRunStep.yml](https://github.com/MicrosoftDocs/MachineLearning-Python-pr/blob/live/AzureML-Docset/stable/docs-ref-autogen/azureml-pipeline-steps/azureml.pipeline.steps.ParallelRunStep.yml)
* Service: **machine-learning**
* Sub-service: **core**
* GitHub Login: @DebFro
* Microsoft Alias: **debfro**

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.