Azure / Azure/MachineLearningNotebooks

'RegressionPipeline' object has no attribute 'forecast' (Azure AutoML - Regression tasks)

Abierto
#1,652 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Jupyter Notebook
Estrellas
4.4k
Forks
2.6k
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

Hi MSFT team,

I have an issue while adapting the code from notebook:
[auto-ml-forecasting-many-models.ipynb](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/automated-machine-learning/forecasting-many-models/auto-ml-forecasting-many-models.ipynb)

My data inference code which failed:
```
mm_parameters = ManyModelsInferenceParameters(
partition_column_names=partition_column_names,
)

inference_steps = AutoMLPipelineBuilder.get_many_models_batch_inference_steps(
experiment=experiment,
inference_data=registered_inference,
node_count=6,
process_count_per_node=32,
compute_target=compute_target,
run_invocation_timeout=600,
output_datastore=output_inference_data_ds,
train_experiment_name="honest_screw_98nvvyvz",
train_run_id="829b32cd-00c6-49ba-bcdc-606503f85338",
inference_pipeline_parameters=mm_parameters,
)

inference_pipeline = Pipeline(ws, steps=inference_steps)

inference_run = experiment.submit(inference_pipeline)
inference_run.wait_for_completion(show_output=False)
```

The driver_log.txt
```
2021/12/02 08:55:27 Got JobInfoJson from env
2021/12/02 08:55:27 Starting App Insight Logger for task: runTaskLet
2021/12/02 08:55:27 Version: 3.0.01788.0003 Branch: .SourceBranch Commit: 5ea6fd0
2021/12/02 08:55:27 Attempt 1 of http call to http://10.0.0.4:16384/sendlogstoartifacts/info
2021/12/02 08:55:27 Send process info logs to master server succeeded
2021/12/02 08:55:27 Attempt 1 of http call to http://10.0.0.4:16384/sendlogstoartifacts/status
2021/12/02 08:55:27 Send process info logs to master server succeeded
[2021-12-02T08:55:27.192932] Entering context manager injector.
[2021-12-02T08:55:27.629925] context_manager_injector.py Command line Options: Namespace(inject=['ProjectPythonPath:context_managers.ProjectPythonPath', 'Dataset:context_managers.Datasets', 'RunHistory:context_managers.RunHistory', 'TrackUserError:context_managers.TrackUserError'], invocation=['driver/amlbi_main.py', '--client_sdk_version', '1.36.0', '--scoring_module_name', 'many_models_inference_driver.py', '--mini_batch_size', '1048576', '--error_threshold', '-1', '--output_action', 'append_row', '--logging_level', 'INFO', '--run_invocation_timeout', '600', '--run_max_try', '3', '--create_snapshot_at_runtime', 'True', '--output', 'DatasetOutputConfig:DT_DS_RESULT_DATA', '--process_count_per_node', '32', '--partition_keys', '["REGION", "SITE_CODE", "CATEGORY", "DISTNAME", "TIME-CLUSTER"]', '--partition_column_names', 'REGION', 'SITE_CODE', 'CATEGORY', 'DISTNAME', 'TIME-CLUSTER', '--input_ds_0', 'partitioned_dataset'])
Script type = None
[2021-12-02T08:55:27.633149] Entering Run History Context Manager.
[2021-12-02T08:55:30.308521] Current directory: /mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c
[2021-12-02T08:55:30.315362] Preparing to call script [driver/amlbi_main.py] with arguments:['--client_sdk_version', '1.36.0', '--scoring_module_name', 'many_models_inference_driver.py', '--mini_batch_size', '1048576', '--error_threshold', '-1', '--output_action', 'append_row', '--logging_level', 'INFO', '--run_invocation_timeout', '600', '--run_max_try', '3', '--create_snapshot_at_runtime', 'True', '--output', '$DT_DS_RESULT_DATA', '--process_count_per_node', '32', '--partition_keys', '["REGION", "SITE_CODE", "CATEGORY", "DISTNAME", "TIME-CLUSTER"]', '--partition_column_names', 'REGION', 'SITE_CODE', 'CATEGORY', 'DISTNAME', 'TIME-CLUSTER', '--input_ds_0', 'partitioned_dataset']
[2021-12-02T08:55:30.315398] After variable expansion, calling script [driver/amlbi_main.py] with arguments:['--client_sdk_version', '1.36.0', '--scoring_module_name', 'many_models_inference_driver.py', '--mini_batch_size', '1048576', '--error_threshold', '-1', '--output_action', 'append_row', '--logging_level', 'INFO', '--run_invocation_timeout', '600', '--run_max_try', '3', '--create_snapshot_at_runtime', 'True', '--output', '/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/DT_DS_RESULT_DATA_workspaceblobstore', '--process_count_per_node', '32', '--partition_keys', '["REGION", "SITE_CODE", "CATEGORY", "DISTNAME", "TIME-CLUSTER"]', '--partition_column_names', 'REGION', 'SITE_CODE', 'CATEGORY', 'DISTNAME', 'TIME-CLUSTER', '--input_ds_0', 'partitioned_dataset']

2021/12/02 08:55:32 Not exporting to RunHistory as the exporter is either stopped or there is no data.
Stopped: false
OriginalData: 1
FilteredData: 0.

[2021-12-02T08:57:21.764141] The experiment failed. Finalizing run...
Cleaning up all outstanding Run operations, waiting 900.0 seconds
4 items cleaning up...
Cleanup took 0.2794077396392822 seconds
azureml_common.parallel_run.exception_info.Exception: Run failed. Below is the error detail:
Traceback (most recent call last):
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/driver/simulator.py", line 93, in main
simulator.wait()
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/masterless_simulator.py", line 128, in wait
JobResult().check_job_result()
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/job_result.py", line 168, in check_job_result
raise EntryScriptException(
DriverException: There is no succeeded mini batch item returned from run(). Please check 'response: run()' in https://aka.ms/batch-inference-documentation
Please check logs for error. You can check logs/readme.txt for the layout of logs. The master role failed with error: There is no succeeded mini batch item returned from run(). Please check 'response: run()' in https://aka.ms/batch-inference-documentation
Please check logs for error. You can check logs/readme.txt for the layout of logs. Detail: Traceback (most recent call last):
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/master_role.py", line 420, in finish_job
self.summarize()
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/master_role.py", line 306, in summarize
raise NoResultToAppendError(msg)
azureml_common.parallel_run.exception.NoResultToAppendError: There is no succeeded mini batch item returned from run(). Please check 'response: run()' in https://aka.ms/batch-inference-documentation
Please check logs for error. You can check logs/readme.txt for the layout of logs.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "driver/amlbi_main.py", line 174, in
main()
File "driver/amlbi_main.py", line 123, in main
boot(driver_dir)
File "driver/amlbi_main.py", line 58, in boot
booter.start()
File "driver/azureml_user/parallel_run/boot.py", line 372, in start
self.start_sys_main()
File "driver/azureml_user/parallel_run/boot.py", line 260, in start_sys_main
self.run_sys_main(cmd)
File "driver/azureml_user/parallel_run/boot_simulator.py", line 40, in run_sys_main
self.run(cmd)
File "driver/azureml_user/parallel_run/boot.py", line 201, in run
self.check_run_result(proc=proc, stdout=stdout, stderr=stderr)
File "driver/azureml_user/parallel_run/boot.py", line 211, in check_run_result
BootResult().check_result(stdout)
File "driver/azureml_user/parallel_run/boot_result.py", line 36, in check_result
raise Exception(message) from cause
Exception: Run failed, please check logs for details. You can check logs/readme.txt for the layout of logs.

[2021-12-02T08:57:22.194362] Finished context manager injector with Exception.
2021/12/02 08:57:24 Skipping parsing control script error. Reason: Error json file doesn't exist. This most likely means that no errors were written to the file. File path: /mnt/batch/tasks/workitems/d39a8c4d-fce8-46ce-ae24-f8807a9db290/job-1/c45fac8e-ddbf-4c65-8_3d0976e1-7980-491d-ab53-a62da09cc329/wd/runTaskLetTask_error.json
2021/12/02 08:57:24 Wrapper cmd failed with err: exit status 1
2021/12/02 08:57:24 Attempt 1 of http call to http://10.0.0.4:16384/sendlogstoartifacts/status
2021/12/02 08:57:24 Send process info logs to master server succeeded
2021/12/02 08:57:24 mpirun version string: {
mpirun (Open MPI) 3.1.2

Report bugs to http://www.open-mpi.org/community/help/
}
2021/12/02 08:57:24 Not exporting to RunHistory as the exporter is either stopped or there is no data.
Stopped: false
OriginalData: 2
FilteredData: 0.
2021/12/02 08:57:24 Process Exiting with Code: 1
2021/12/02 08:57:24 All App Insights Logs was sent successfully or the close timeout of 10 was reached
```

The job_error.xxxxxx.txt
```
2021-12-02T08:55:38.915706 This file shows errors in loading, calling init() and calling run() of the entry script. An error here may not impact the job result. Please check "job_result.txt" for the job result.
2021-12-02T08:56:09.734605 The process017 on node 10.0.0.4 failed on mini batch 2 with error: Traceback (most recent call last):
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/driver/azureml_user/parallel_run/payload_thread.py", line 280, in process_task
self.process_mini_batch(task)
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/driver/azureml_user/parallel_run/payload_thread.py", line 108, in process_mini_batch
score_output = processor.process_mini_batch(*args)
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/driver/azureml_user/parallel_run/task_processor.py", line 247, in process_mini_batch
score_output = score_module.call_run(inputs, mini_batch_context)
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/driver/azureml_common/parallel_run/stderr_converter.py", line 29, in wrapper
res = method(self, *args, **kwargs)
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/driver/azureml_user/parallel_run/score_module.py", line 235, in call_run
return self.run(*args)
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/many_models_inference_driver.py", line 51, in run
return many_models_inference.run(input_data)
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/many_models_inference.py", line 55, in run
all_predictions = super(ManyModelsInference, self).run(input_data)
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/automl_prs_run_base.py", line 183, in run
self._run_subprocess_for_file(input_data_file, output_file)
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/automl_prs_run_base.py", line 220, in _run_subprocess_for_file
"stderr from subprocess: \n{}\n".format(subprocess.returncode, subprocess_stderr))
Exception: AutoML training subprocess exited unsuccessfully with error code: 1
stderr from subprocess:
Traceback (most recent call last):
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/run_prs_driver.py", line 41, in
run_prs_driver()
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/run_prs_driver.py", line 36, in run_prs_driver
logs = prs_driver.run(data_file_path, output_file)
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/many_models_inference_driver.py", line 60, in run
data = self._do_inference(data)
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/many_models_inference_driver.py", line 129, in _do_inference
y_predictions, X_trans = model.forecast(X_test, ignore_data_errors=True)
AttributeError: 'RegressionPipeline' object has no attribute 'forecast'
```

My task configuration (AutoML):
```
--client_sdk_version 1.36.0 --scoring_module_name many_models_inference_driver.py --mini_batch_size 1048576 --error_threshold -1 --output_action append_row --logging_level INFO --run_invocation_timeout 600 --run_max_try 3 --create_snapshot_at_runtime True --output DatasetOutputConfig:DT_DS_RESULT_DATA --process_count_per_node 32 --partition_keys ["REGION", "SITE_CODE", "CATEGORY", "DISTNAME", "TIME-CLUSTER"] --partition_column_names REGION SITE_CODE CATEGORY DISTNAME TIME-CLUSTER --input_ds_0 partitioned_dataset
```
As you can see, the error is **'RegressionPipeline' object has no attribute 'forecast'**.
What should I do to fix this?
Thank you,

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.