Azure / Azure/azureml-examples
AzureMLException: Message: Failed to flush task queue within 300.0 seconds | dreambooth finetune text to image
- Dominant language
- Jupyter Notebook
- Stars
- 2k
- Forks
- 1.7k
- Avg merge
- 18h 18m
- Merged PRs (30d)
- 2
Description
### Operating System
Linux
### Version Information
Running the sample in Compute Instance within the Azure ML workspace.
Running the sample provided in azureml-examples
pipeline component : diffusers_text_to_image_dreambooth_pipeline
### Steps to reproduce
1. Ran the notebook as -is provided in azureml-examples "https://github.com/Azure/azureml-examples/blob/main/sdk/python/foundation-models/system/finetune/text-to-image/diffusers-dreambooth-dog-text-to-image.ipynb"
### Expected behavior
Finetune the model successfully.
### Actual behavior
encountered error in the child job "text_to_image_dreambooth_finetune".
Encountered an internal ACFT error. Error Message/Code: AzureMLException:
Message: Failed to flush task queue within 300.0 seconds. Please set AZUREML_ARTIFACTS_DEFAULT_TIMEOUT environment variable to increase the timeout(in seconds)
InnerException None
ErrorResponse
{
"error": {
"code": "UserError",
"message": "Failed to flush task queue within 300.0 seconds. Please set AZUREML_ARTIFACTS_DEFAULT_TIMEOUT environment variable to increase the timeout(in seconds) ",
"inner_error": {
"code": "ResourceExhausted",
"inner_error": {
"code": "Timeout"
}
}
}
}. Traceback: File "swallow_all_exceptions_decorator.py", line 68, in wrapper
return func(*args, **kwargs)
File "[Non-AutoML file]", line 1097, in [Non-AutoML function]
File "finetune_runner.py", line 252, in finetune_runner
mlflow.log_artifacts(component_args.output_dir, SettingParameters.DEFAULT_OUTPUT_DIR)
File "[Non-AutoML file]", line 1096, in [Non-AutoML function]
File "[Non-AutoML file]", line 1242, in [Non-AutoML function]
File "[Non-AutoML file]", line 570, in [Non-AutoML function]
File "[Non-AutoML file]", line 88, in [Non-AutoML function]
File "[Non-AutoML file]", line 97, in [Non-AutoML function]
File "[Non-AutoML file]", line 29, in [Non-AutoML function]
File "[Non-AutoML file]", line 55, in [Non-AutoML function]
File "[Non-AutoML file]", line 135, in [Non-AutoML function]
. Additional information: [Hidden as it may contain PII].
4.
### Addition information
Talks about increasing the timeout. Not sure in compute instance how to find the file and change the default time out.
Contributor guide
Research direction
Start with the linked diffusers-dreambooth-dog-text-to-image.ipynb sample and the diffusers_text_to_image_dreambooth_pipeline child job. Trace the reported failure through finetune_runner.py at mlflow.log_artifacts, then determine where AZUREML_ARTIFACTS_DEFAULT_TIMEOUT can be configured for a Compute Instance. Done means the sample finetunes successfully without the task-queue timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, jupyter-notebook, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100