apache / apache/cloudstack

VR healthcheck wrong state

Đang mở
#12,630 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
component:healthcheck
Ngôn ngữ chính
Java
Star
3.1k
Fork
1.4k
Merge trung bình
6 ngày 19 giờ
Pull request đã merge (30 ngày)
32

Mô tả

### problem

The listRouters API-call always returns healthcheckfailed false, even if the healthcheck fails.

Alert in CloudStack UI:

```
Health checks failed: 1 failing checks on router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16
```

Error in EventQueue

```
{"eventDateTime":"2026-02-09 14:01:22 +0100","subject":"Health checks failed: 1 failing checks on router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16","dataCenterId":"a8f95837-c05c-4a97-b100-53f87de8df24","event":"ALERT.SERVICE.DOMAINROUTER","body":"Health checks failed: 1 failing checks on router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16","podId":"090ccb90-9411-4bab-bb0c-f5ab61e34ebb"}
{"eventDateTime":"2026-02-09 14:01:22 +0100","entityuuid":"c0c6a15f-3ad8-4936-ba6e-abe802ddba16","description":"Router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16 has failing checks: router_version_check.py","event":"ROUTER.HEALTH.CHECKS","user":"c302beae-cb9d-11f0-a957-fa163ed04c4b","account":"c302a9b2-cb9d-11f0-a957-fa163ed04c4b","entity":"DomainRouter","status":"Completed"}
```

API call via cmk

```
(cloudstackmsu) 🐄 > list routers name=r-8-VM filter=name,hostname,healthchecksfailed,healthcheckresults,state, fetchhealthcheckresults=true listall=true
router:
count = 1
+--------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+
| NAME | HOSTNAME | HEALTHCHECKSFAILED | HEALTHCHECKRESULTS | STATE |
+--------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+
| r-8-VM | srv1-214 | false | [{"checkname":"connectivity.test","checktype":"basic","details":"Failed to fetch results with details: java.io.IOException: Stream closed\n\tat | Running |
| | | | java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)\n\tat java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)\n\tat | |
| | | | java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)\n\tat java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)\n\tat | |
| | | | java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)\n\tat java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)\n\tat | |
| | | | java.base/java.io.BufferedReader.fill(BufferedReader.java:162)\n\tat java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)\n\tat | |
| | | | java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)\n\tat com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)\n\tat | |
| | | | com.cloud.utils.script.Script.execute(Script.java:336)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:632)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:620)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:431)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:403)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:239)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:130)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)\n\tat | |
| | | | com.cloud.agent.Agent.processRequest(Agent.java:813)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)\n\tat | |
| | | | com.cloud.utils.nio.Task.call(Task.java:83)\n\tat com.cloud.utils.nio.Task.call(Task.java:29)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat | |
| | | | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat | |
| | | | java.base/java.lang.Thread.run(Thread.java:840)\n","lastupdated":"2026-02-11T13:01:46+0100","status":"FAILED","success":false},{"checkname":"filesystem.writable.test","checktype":"basic","details":"Failed | |
| | | | to fetch results with details: java.io.IOException: Stream closed\n\tat java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)\n\tat | |
| | | | java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)\n\tat java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)\n\tat | |
| | | | java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)\n\tat java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)\n\tat | |
| | | | java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)\n\tat java.base/java.io.BufferedReader.fill(BufferedReader.java:162)\n\tat | |
| | | | java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)\n\tat java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)\n\tat | |
| | | | com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)\n\tat com.cloud.utils.script.Script.execute(Script.java:336)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:632)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:620)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:431)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:403)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:239)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:130)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)\n\tat | |
| | | | com.cloud.agent.Agent.processRequest(Agent.java:813)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)\n\tat | |
| | | | com.cloud.utils.nio.Task.call(Task.java:83)\n\tat com.cloud.utils.nio.Task.call(Task.java:29)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat | |
| | | | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat | |
| | | | java.base/java.lang.Thread.run(Thread.java:840)\n","lastupdated":"2026-02-11T13:01:46+0100","status":"UNKNOWN","success":false}] | |
+--------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+
```

Expected result: healthcheckfailed=true instead of healthcheckfailed=false

### versions

ACS 4.22.0

### The steps to reproduce the bug

1. ssh into one VR VM
2. shutdown the interfaces e.g. `ifdown eth0 ; ifdown eth1`
3. Wait about 10 minutes
4. Check Alerts in the UI
5. Check `cmk list routers filter=name,hostname,healthchecksfailed,healthcheckresults,state, fetchhealthcheckresults=true listall=true`

### What to do about it?

_No response_

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Bắt đầu tại VirtualRoutingResource.checkRouterFileSystem và theo dõi cách các kết quả health-check thất bại đi qua KVM agent vào phản hồi API listRouters. Tái hiện trường hợp bằng cách sử dụng fetchhealthcheckresults=true và so sánh healthchecksfailed với các kết quả FAILED được báo cáo; hoàn tất khi API báo cáo trạng thái thất bại nhất quán với các kết quả đó.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
java, python
Lĩnh vực
backend, infrastructure
Loại issue
Lỗi
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Đình trệ
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
35/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.