Azure / Azure/MachineLearningNotebooks
'RegressionPipeline' object has no attribute 'forecast' (Azure AutoML - Regression tasks)
- Dominant language
- Jupyter Notebook
- Stars
- 4.4k
- Forks
- 2.6k
- PR merge metrics
- No merged PRs in 30d
Description
Hi MSFT team,
I have an issue while adapting the code from notebook:
[auto-ml-forecasting-many-models.ipynb](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/automated-machine-learning/forecasting-many-models/auto-ml-forecasting-many-models.ipynb)
My data inference code which failed:
```
mm_parameters = ManyModelsInferenceParameters(
partition_column_names=partition_column_names,
)
inference_steps = AutoMLPipelineBuilder.get_many_models_batch_inference_steps(
experiment=experiment,
inference_data=registered_inference,
node_count=6,
process_count_per_node=32,
compute_target=compute_target,
run_invocation_timeout=600,
output_datastore=output_inference_data_ds,
train_experiment_name="honest_screw_98nvvyvz",
train_run_id="829b32cd-00c6-49ba-bcdc-606503f85338",
inference_pipeline_parameters=mm_parameters,
)
inference_pipeline = Pipeline(ws, steps=inference_steps)
inference_run = experiment.submit(inference_pipeline)
inference_run.wait_for_completion(show_output=False)
```
The driver_log.txt
```
2021/12/02 08:55:27 Got JobInfoJson from env
2021/12/02 08:55:27 Starting App Insight Logger for task: runTaskLet
2021/12/02 08:55:27 Version: 3.0.01788.0003 Branch: .SourceBranch Commit: 5ea6fd0
2021/12/02 08:55:27 Attempt 1 of http call to http://10.0.0.4:16384/sendlogstoartifacts/info
2021/12/02 08:55:27 Send process info logs to master server succeeded
2021/12/02 08:55:27 Attempt 1 of http call to http://10.0.0.4:16384/sendlogstoartifacts/status
2021/12/02 08:55:27 Send process info logs to master server succeeded
[2021-12-02T08:55:27.192932] Entering context manager injector.
[2021-12-02T08:55:27.629925] context_manager_injector.py Command line Options: Namespace(inject=['ProjectPythonPath:context_managers.ProjectPythonPath', 'Dataset:context_managers.Datasets', 'RunHistory:context_managers.RunHistory', 'TrackUserError:context_managers.TrackUserError'], invocation=['driver/amlbi_main.py', '--client_sdk_version', '1.36.0', '--scoring_module_name', 'many_models_inference_driver.py', '--mini_batch_size', '1048576', '--error_threshold', '-1', '--output_action', 'append_row', '--logging_level', 'INFO', '--run_invocation_timeout', '600', '--run_max_try', '3', '--create_snapshot_at_runtime', 'True', '--output', 'DatasetOutputConfig:DT_DS_RESULT_DATA', '--process_count_per_node', '32', '--partition_keys', '["REGION", "SITE_CODE", "CATEGORY", "DISTNAME", "TIME-CLUSTER"]', '--partition_column_names', 'REGION', 'SITE_CODE', 'CATEGORY', 'DISTNAME', 'TIME-CLUSTER', '--input_ds_0', 'partitioned_dataset'])
Script type = None
[2021-12-02T08:55:27.633149] Entering Run History Context Manager.
[2021-12-02T08:55:30.308521] Current directory: /mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c
[2021-12-02T08:55:30.315362] Preparing to call script [driver/amlbi_main.py] with arguments:['--client_sdk_version', '1.36.0', '--scoring_module_name', 'many_models_inference_driver.py', '--mini_batch_size', '1048576', '--error_threshold', '-1', '--output_action', 'append_row', '--logging_level', 'INFO', '--run_invocation_timeout', '600', '--run_max_try', '3', '--create_snapshot_at_runtime', 'True', '--output', '$DT_DS_RESULT_DATA', '--process_count_per_node', '32', '--partition_keys', '["REGION", "SITE_CODE", "CATEGORY", "DISTNAME", "TIME-CLUSTER"]', '--partition_column_names', 'REGION', 'SITE_CODE', 'CATEGORY', 'DISTNAME', 'TIME-CLUSTER', '--input_ds_0', 'partitioned_dataset']
[2021-12-02T08:55:30.315398] After variable expansion, calling script [driver/amlbi_main.py] with arguments:['--client_sdk_version', '1.36.0', '--scoring_module_name', 'many_models_inference_driver.py', '--mini_batch_size', '1048576', '--error_threshold', '-1', '--output_action', 'append_row', '--logging_level', 'INFO', '--run_invocation_timeout', '600', '--run_max_try', '3', '--create_snapshot_at_runtime', 'True', '--output', '/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/DT_DS_RESULT_DATA_workspaceblobstore', '--process_count_per_node', '32', '--partition_keys', '["REGION", "SITE_CODE", "CATEGORY", "DISTNAME", "TIME-CLUSTER"]', '--partition_column_names', 'REGION', 'SITE_CODE', 'CATEGORY', 'DISTNAME', 'TIME-CLUSTER', '--input_ds_0', 'partitioned_dataset']
2021/12/02 08:55:32 Not exporting to RunHistory as the exporter is either stopped or there is no data.
Stopped: false
OriginalData: 1
FilteredData: 0.
[2021-12-02T08:57:21.764141] The experiment failed. Finalizing run...
Cleaning up all outstanding Run operations, waiting 900.0 seconds
4 items cleaning up...
Cleanup took 0.2794077396392822 seconds
azureml_common.parallel_run.exception_info.Exception: Run failed. Below is the error detail:
Traceback (most recent call last):
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/driver/simulator.py", line 93, in main
simulator.wait()
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/masterless_simulator.py", line 128, in wait
JobResult().check_job_result()
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/job_result.py", line 168, in check_job_result
raise EntryScriptException(
DriverException: There is no succeeded mini batch item returned from run(). Please check 'response: run()' in https://aka.ms/batch-inference-documentation
Please check logs for error. You can check logs/readme.txt for the layout of logs. The master role failed with error: There is no succeeded mini batch item returned from run(). Please check 'response: run()' in https://aka.ms/batch-inference-documentation
Please check logs for error. You can check logs/readme.txt for the layout of logs. Detail: Traceback (most recent call last):
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/master_role.py", line 420, in finish_job
self.summarize()
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/prs_prod/lib/python3.8/site-packages/azureml_sys/parallel_run/master_role.py", line 306, in summarize
raise NoResultToAppendError(msg)
azureml_common.parallel_run.exception.NoResultToAppendError: There is no succeeded mini batch item returned from run(). Please check 'response: run()' in https://aka.ms/batch-inference-documentation
Please check logs for error. You can check logs/readme.txt for the layout of logs.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "driver/amlbi_main.py", line 174, in
main()
File "driver/amlbi_main.py", line 123, in main
boot(driver_dir)
File "driver/amlbi_main.py", line 58, in boot
booter.start()
File "driver/azureml_user/parallel_run/boot.py", line 372, in start
self.start_sys_main()
File "driver/azureml_user/parallel_run/boot.py", line 260, in start_sys_main
self.run_sys_main(cmd)
File "driver/azureml_user/parallel_run/boot_simulator.py", line 40, in run_sys_main
self.run(cmd)
File "driver/azureml_user/parallel_run/boot.py", line 201, in run
self.check_run_result(proc=proc, stdout=stdout, stderr=stderr)
File "driver/azureml_user/parallel_run/boot.py", line 211, in check_run_result
BootResult().check_result(stdout)
File "driver/azureml_user/parallel_run/boot_result.py", line 36, in check_result
raise Exception(message) from cause
Exception: Run failed, please check logs for details. You can check logs/readme.txt for the layout of logs.
[2021-12-02T08:57:22.194362] Finished context manager injector with Exception.
2021/12/02 08:57:24 Skipping parsing control script error. Reason: Error json file doesn't exist. This most likely means that no errors were written to the file. File path: /mnt/batch/tasks/workitems/d39a8c4d-fce8-46ce-ae24-f8807a9db290/job-1/c45fac8e-ddbf-4c65-8_3d0976e1-7980-491d-ab53-a62da09cc329/wd/runTaskLetTask_error.json
2021/12/02 08:57:24 Wrapper cmd failed with err: exit status 1
2021/12/02 08:57:24 Attempt 1 of http call to http://10.0.0.4:16384/sendlogstoartifacts/status
2021/12/02 08:57:24 Send process info logs to master server succeeded
2021/12/02 08:57:24 mpirun version string: {
mpirun (Open MPI) 3.1.2
Report bugs to http://www.open-mpi.org/community/help/
}
2021/12/02 08:57:24 Not exporting to RunHistory as the exporter is either stopped or there is no data.
Stopped: false
OriginalData: 2
FilteredData: 0.
2021/12/02 08:57:24 Process Exiting with Code: 1
2021/12/02 08:57:24 All App Insights Logs was sent successfully or the close timeout of 10 was reached
```
The job_error.xxxxxx.txt
```
2021-12-02T08:55:38.915706 This file shows errors in loading, calling init() and calling run() of the entry script. An error here may not impact the job result. Please check "job_result.txt" for the job result.
2021-12-02T08:56:09.734605 The process017 on node 10.0.0.4 failed on mini batch 2 with error: Traceback (most recent call last):
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/driver/azureml_user/parallel_run/payload_thread.py", line 280, in process_task
self.process_mini_batch(task)
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/driver/azureml_user/parallel_run/payload_thread.py", line 108, in process_mini_batch
score_output = processor.process_mini_batch(*args)
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/driver/azureml_user/parallel_run/task_processor.py", line 247, in process_mini_batch
score_output = score_module.call_run(inputs, mini_batch_context)
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/driver/azureml_common/parallel_run/stderr_converter.py", line 29, in wrapper
res = method(self, *args, **kwargs)
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/driver/azureml_user/parallel_run/score_module.py", line 235, in call_run
return self.run(*args)
File "/mnt/batch/tasks/shared/LS_root/jobs/bnlwe-da16-d-901557-aml-01/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/wd/azureml/c45fac8e-ddbf-4c65-84d9-a1d1006d856c/many_models_inference_driver.py", line 51, in run
return many_models_inference.run(input_data)
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/many_models_inference.py", line 55, in run
all_predictions = super(ManyModelsInference, self).run(input_data)
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/automl_prs_run_base.py", line 183, in run
self._run_subprocess_for_file(input_data_file, output_file)
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/automl_prs_run_base.py", line 220, in _run_subprocess_for_file
"stderr from subprocess: \n{}\n".format(subprocess.returncode, subprocess_stderr))
Exception: AutoML training subprocess exited unsuccessfully with error code: 1
stderr from subprocess:
Traceback (most recent call last):
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/run_prs_driver.py", line 41, in
run_prs_driver()
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/run_prs_driver.py", line 36, in run_prs_driver
logs = prs_driver.run(data_file_path, output_file)
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/many_models_inference_driver.py", line 60, in run
data = self._do_inference(data)
File "/azureml-envs/azureml_705720c76ff57b57c77d577152dabb18/lib/python3.6/site-packages/azureml/train/automl/runtime/_many_models/many_models_inference_driver.py", line 129, in _do_inference
y_predictions, X_trans = model.forecast(X_test, ignore_data_errors=True)
AttributeError: 'RegressionPipeline' object has no attribute 'forecast'
```
My task configuration (AutoML):
```
--client_sdk_version 1.36.0 --scoring_module_name many_models_inference_driver.py --mini_batch_size 1048576 --error_threshold -1 --output_action append_row --logging_level INFO --run_invocation_timeout 600 --run_max_try 3 --create_snapshot_at_runtime True --output DatasetOutputConfig:DT_DS_RESULT_DATA --process_count_per_node 32 --partition_keys ["REGION", "SITE_CODE", "CATEGORY", "DISTNAME", "TIME-CLUSTER"] --partition_column_names REGION SITE_CODE CATEGORY DISTNAME TIME-CLUSTER --input_ds_0 partitioned_dataset
```
As you can see, the error is **'RegressionPipeline' object has no attribute 'forecast'**.
What should I do to fix this?
Thank you,
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.