Azure / Azure/MachineLearningNotebooks
Service deployment polling reached non-successful terminal state, current service state: Unhealthy
- Linguagem predominante
- Jupyter Notebook
- Estrelas
- 4.4k
- Forks
- 2.6k
- Métricas de merge de PRs
- Nenhum PR com merge em 30d
Descrição
Hi,
When I mapped an SSL certificate to Azure ML endpoint with AKS compute target using the CLI commands It is successful at the time End point status is get in an Unhealthy state. we using our custom DNS and custom certificates. The cluster pods, nodes, and deployments all are in a running state and deployment was successful but the endpoint an Unhealthy state. So help to fix the issue.
Error:
Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c
More information can be found using '.get_logs()'
Error:
{
"code": "DeploymentTimedOut",
"statusCode": 504,
"message": "Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087"
}
Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c
More information can be found using '.get_logs()'
Error:
{
"code": "DeploymentTimedOut",
"statusCode": 504,
"message": "Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087"
}
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/azureml/core/webservice/webservice.py", line 671, in wait_for_deployment
raise WebserviceException('Service deployment polling reached non-successful terminal state, current '
azureml.exceptions._azureml_exception.WebserviceException: WebserviceException:
Message: Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c
More information can be found using '.get_logs()'
Error:
{
"code": "DeploymentTimedOut",
"statusCode": 504,
"message": "Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087"
}
InnerException None
ErrorResponse
{
"error": {
"message": "Service deployment polling reached non-successful terminal state, current service state: Unhealthy\nOperation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c\nMore information can be found using '.get_logs()'\nError:\n{\n \"code\": \"DeploymentTimedOut\",\n \"statusCode\": 504,\n \"message\": \"Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087\"\n}"
}
}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3437, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 157, in
service.wait_for_deployment(show_output=True)
File "/usr/local/lib/python3.8/site-packages/azureml/core/webservice/webservice.py", line 686, in wait_for_deployment
raise WebserviceException(e.message, logger=module_logger)
azureml.exceptions._azureml_exception.WebserviceException: WebserviceException:
Message: Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c
More information can be found using '.get_logs()'
Error:
{
"code": "DeploymentTimedOut",
"statusCode": 504,
"message": "Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087"
}
InnerException None
ErrorResponse
{
"error": {
"message": "Service deployment polling reached non-successful terminal state, current service state: Unhealthy\nOperation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c\nMore information can be found using '.get_logs()'\nError:\n{\n \"code\": \"DeploymentTimedOut\",\n \"statusCode\": 504,\n \"message\": \"Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087\"\n}"
}
}
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Direção de pesquisa
Comece com a chamada service.wait_for_deployment() e a saída correspondente de .get_logs(); em seguida, revise como o deployment do AKS, o DNS personalizado e o certificado personalizado estão configurados. A issue estará concluída quando a causa do estado terminal Unhealthy for identificada e uma correção de deployment reproduzível ou uma limitação documentada for estabelecida.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- azure, kubernetes, python
- Domínio
- cloud, devops, infrastructure, machine-learning
- Tipo de issue
- Bug
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Status de atividade
- Estagnada
- Clareza
- Precisa de esclarecimento
- Facilidade para iniciantes
- 20/100