Azure / Azure/MachineLearningNotebooks
Service deployment polling reached non-successful terminal state, current service state: Unhealthy
- 主要言語
- Jupyter Notebook
- スター
- 4.4k
- フォーク
- 2.6k
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
Hi,
When I mapped an SSL certificate to Azure ML endpoint with AKS compute target using the CLI commands It is successful at the time End point status is get in an Unhealthy state. we using our custom DNS and custom certificates. The cluster pods, nodes, and deployments all are in a running state and deployment was successful but the endpoint an Unhealthy state. So help to fix the issue.
Error:
Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c
More information can be found using '.get_logs()'
Error:
{
"code": "DeploymentTimedOut",
"statusCode": 504,
"message": "Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087"
}
Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c
More information can be found using '.get_logs()'
Error:
{
"code": "DeploymentTimedOut",
"statusCode": 504,
"message": "Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087"
}
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/azureml/core/webservice/webservice.py", line 671, in wait_for_deployment
raise WebserviceException('Service deployment polling reached non-successful terminal state, current '
azureml.exceptions._azureml_exception.WebserviceException: WebserviceException:
Message: Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c
More information can be found using '.get_logs()'
Error:
{
"code": "DeploymentTimedOut",
"statusCode": 504,
"message": "Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087"
}
InnerException None
ErrorResponse
{
"error": {
"message": "Service deployment polling reached non-successful terminal state, current service state: Unhealthy\nOperation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c\nMore information can be found using '.get_logs()'\nError:\n{\n \"code\": \"DeploymentTimedOut\",\n \"statusCode\": 504,\n \"message\": \"Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087\"\n}"
}
}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3437, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 157, in
service.wait_for_deployment(show_output=True)
File "/usr/local/lib/python3.8/site-packages/azureml/core/webservice/webservice.py", line 686, in wait_for_deployment
raise WebserviceException(e.message, logger=module_logger)
azureml.exceptions._azureml_exception.WebserviceException: WebserviceException:
Message: Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c
More information can be found using '.get_logs()'
Error:
{
"code": "DeploymentTimedOut",
"statusCode": 504,
"message": "Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087"
}
InnerException None
ErrorResponse
{
"error": {
"message": "Service deployment polling reached non-successful terminal state, current service state: Unhealthy\nOperation ID: 20dee957-d034-4104-8b67-8e472e6b6a6c\nMore information can be found using '.get_logs()'\nError:\n{\n \"code\": \"DeploymentTimedOut\",\n \"statusCode\": 504,\n \"message\": \"Timed out waiting for AKS deployment to complete. pollTimeout : 00:20:00 serviceName: oneforma-ocr-gpu-real-time serviceId: f94a68c5-8cf3-4532-a04d-c1adc7a48087\"\n}"
}
}
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
service.wait_for_deployment() の呼び出しと、それに対応する .get_logs() の出力から始め、AKS デプロイ、カスタム DNS、カスタム証明書がどのように構成されているかを確認します。Unhealthy の終端状態の原因が特定され、再現可能なデプロイの修正または文書化された制限が確立された時点で、issue は完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- azure, kubernetes, python
- 領域
- cloud, devops, infrastructure, machine-learning
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 20/100