Azure / Azure/azureml-examples

AzureMLException: Message: Failed to flush task queue within 300.0 seconds | dreambooth finetune text to image

Open
#3,313 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

### Operating System

Linux

### Version Information

Running the sample in Compute Instance within the Azure ML workspace.
Running the sample provided in azureml-examples
pipeline component : diffusers_text_to_image_dreambooth_pipeline

### Steps to reproduce

1. Ran the notebook as -is provided in azureml-examples "https://github.com/Azure/azureml-examples/blob/main/sdk/python/foundation-models/system/finetune/text-to-image/diffusers-dreambooth-dog-text-to-image.ipynb"

### Expected behavior

Finetune the model successfully.

### Actual behavior

encountered error in the child job "text_to_image_dreambooth_finetune".

Encountered an internal ACFT error. Error Message/Code: AzureMLException:
Message: Failed to flush task queue within 300.0 seconds. Please set AZUREML_ARTIFACTS_DEFAULT_TIMEOUT environment variable to increase the timeout(in seconds)
InnerException None
ErrorResponse
{
"error": {
"code": "UserError",
"message": "Failed to flush task queue within 300.0 seconds. Please set AZUREML_ARTIFACTS_DEFAULT_TIMEOUT environment variable to increase the timeout(in seconds) ",
"inner_error": {
"code": "ResourceExhausted",
"inner_error": {
"code": "Timeout"
}
}
}
}. Traceback: File "swallow_all_exceptions_decorator.py", line 68, in wrapper
return func(*args, **kwargs)
File "[Non-AutoML file]", line 1097, in [Non-AutoML function]
File "finetune_runner.py", line 252, in finetune_runner
mlflow.log_artifacts(component_args.output_dir, SettingParameters.DEFAULT_OUTPUT_DIR)
File "[Non-AutoML file]", line 1096, in [Non-AutoML function]
File "[Non-AutoML file]", line 1242, in [Non-AutoML function]
File "[Non-AutoML file]", line 570, in [Non-AutoML function]
File "[Non-AutoML file]", line 88, in [Non-AutoML function]
File "[Non-AutoML file]", line 97, in [Non-AutoML function]
File "[Non-AutoML file]", line 29, in [Non-AutoML function]
File "[Non-AutoML file]", line 55, in [Non-AutoML function]
File "[Non-AutoML file]", line 135, in [Non-AutoML function]
. Additional information: [Hidden as it may contain PII].
4.

### Addition information

Talks about increasing the timeout. Not sure in compute instance how to find the file and change the default time out.

Contributor guide

Open the contributing guide

Research direction

Start with the linked diffusers-dreambooth-dog-text-to-image.ipynb sample and the diffusers_text_to_image_dreambooth_pipeline child job. Trace the reported failure through finetune_runner.py at mlflow.log_artifacts, then determine where AZUREML_ARTIFACTS_DEFAULT_TIMEOUT can be configured for a Compute Instance. Done means the sample finetunes successfully without the task-queue timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.