Azure / Azure/MachineLearningNotebooks

Service deployment polling reached non-successful terminal state, current service state: Unhealthy

未關閉
#1,767 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Jupyter Notebook
星號
4.4k
分支
2.6k
PR 合併指標
30 天內沒有已合併 PR

描述

Hi,

When I mapped an SSL certificate to Azure ML endpoint with AKS compute target using the CLI commands It is successful at the time End point status is get in an Unhealthy state. we using our custom DNS and custom certificates. The cluster pods, nodes, and deployments all are in a running state and deployment was successful but the endpoint an Unhealthy state. So help to fix the issue.

Error:
Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c
More information can be found using '.get_logs()'
Error:
{
"code": "DeploymentTimedOut",
"statusCode": 504,
"message": "Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087"
}
Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c
More information can be found using '.get_logs()'
Error:
{
"code": "DeploymentTimedOut",
"statusCode": 504,
"message": "Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087"
}
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/azureml/core/webservice/webservice.py", line 671, in wait_for_deployment
raise WebserviceException('Service deployment polling reached non-successful terminal state, current '
azureml.exceptions._azureml_exception.WebserviceException: WebserviceException:
Message: Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c
More information can be found using '.get_logs()'
Error:
{
"code": "DeploymentTimedOut",
"statusCode": 504,
"message": "Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087"
}
InnerException None
ErrorResponse
{
"error": {
"message": "Service deployment polling reached non-successful terminal state, current service state: Unhealthy\nOperation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c\nMore information can be found using '.get_logs()'\nError:\n{\n \"code\": \"DeploymentTimedOut\",\n \"statusCode\": 504,\n \"message\": \"Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087\"\n}"
}
}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3437, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 157, in
service.wait_for_deployment(show_output=True)
File "/usr/local/lib/python3.8/site-packages/azureml/core/webservice/webservice.py", line 686, in wait_for_deployment
raise WebserviceException(e.message, logger=module_logger)
azureml.exceptions._azureml_exception.WebserviceException: WebserviceException:
Message: Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c
More information can be found using '.get_logs()'
Error:
{
"code": "DeploymentTimedOut",
"statusCode": 504,
"message": "Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087"
}
InnerException None
ErrorResponse
{
"error": {
"message": "Service deployment polling reached non-successful terminal state, current service state: Unhealthy\nOperation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c\nMore information can be found using '.get_logs()'\nError:\n{\n \"code\": \"DeploymentTimedOut\",\n \"statusCode\": 504,\n \"message\": \"Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087\"\n}"
}
}

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

從 service.wait_for_deployment() 呼叫及其對應的 .get_logs() 輸出開始,然後檢視 AKS 部署、自訂 DNS 和自訂憑證的設定方式。當確定 Unhealthy 終端狀態的原因,並建立可重現的部署修正方案或記錄在案的限制時,該 issue 即完成。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
azure, kubernetes, python
領域
cloud, devops, infrastructure, machine-learning
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
需要釐清
新手友好度
20/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。