VR healthcheck wrong state
- Vorherrschende Sprache
- Java
- Sterne
- 3.1k
- Forks
- 1.4k
- Ø Merge
- 6 T. 19 Std.
- Gemergte PRs (30 T.)
- 32
Beschreibung
### problem
The listRouters API-call always returns healthcheckfailed false, even if the healthcheck fails.
Alert in CloudStack UI:
```
Health checks failed: 1 failing checks on router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16
```
Error in EventQueue
```
{"eventDateTime":"2026-02-09 14:01:22 +0100","subject":"Health checks failed: 1 failing checks on router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16","dataCenterId":"a8f95837-c05c-4a97-b100-53f87de8df24","event":"ALERT.SERVICE.DOMAINROUTER","body":"Health checks failed: 1 failing checks on router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16","podId":"090ccb90-9411-4bab-bb0c-f5ab61e34ebb"}
{"eventDateTime":"2026-02-09 14:01:22 +0100","entityuuid":"c0c6a15f-3ad8-4936-ba6e-abe802ddba16","description":"Router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16 has failing checks: router_version_check.py","event":"ROUTER.HEALTH.CHECKS","user":"c302beae-cb9d-11f0-a957-fa163ed04c4b","account":"c302a9b2-cb9d-11f0-a957-fa163ed04c4b","entity":"DomainRouter","status":"Completed"}
```
API call via cmk
```
(cloudstackmsu) 🐄 > list routers name=r-8-VM filter=name,hostname,healthchecksfailed,healthcheckresults,state, fetchhealthcheckresults=true listall=true
router:
count = 1
+--------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+
| NAME | HOSTNAME | HEALTHCHECKSFAILED | HEALTHCHECKRESULTS | STATE |
+--------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+
| r-8-VM | srv1-214 | false | [{"checkname":"connectivity.test","checktype":"basic","details":"Failed to fetch results with details: java.io.IOException: Stream closed\n\tat | Running |
| | | | java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)\n\tat java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)\n\tat | |
| | | | java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)\n\tat java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)\n\tat | |
| | | | java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)\n\tat java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)\n\tat | |
| | | | java.base/java.io.BufferedReader.fill(BufferedReader.java:162)\n\tat java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)\n\tat | |
| | | | java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)\n\tat com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)\n\tat | |
| | | | com.cloud.utils.script.Script.execute(Script.java:336)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:632)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:620)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:431)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:403)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:239)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:130)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)\n\tat | |
| | | | com.cloud.agent.Agent.processRequest(Agent.java:813)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)\n\tat | |
| | | | com.cloud.utils.nio.Task.call(Task.java:83)\n\tat com.cloud.utils.nio.Task.call(Task.java:29)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat | |
| | | | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat | |
| | | | java.base/java.lang.Thread.run(Thread.java:840)\n","lastupdated":"2026-02-11T13:01:46+0100","status":"FAILED","success":false},{"checkname":"filesystem.writable.test","checktype":"basic","details":"Failed | |
| | | | to fetch results with details: java.io.IOException: Stream closed\n\tat java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)\n\tat | |
| | | | java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)\n\tat java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)\n\tat | |
| | | | java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)\n\tat java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)\n\tat | |
| | | | java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)\n\tat java.base/java.io.BufferedReader.fill(BufferedReader.java:162)\n\tat | |
| | | | java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)\n\tat java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)\n\tat | |
| | | | com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)\n\tat com.cloud.utils.script.Script.execute(Script.java:336)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:632)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:620)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:431)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:403)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:239)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:130)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)\n\tat | |
| | | | com.cloud.agent.Agent.processRequest(Agent.java:813)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)\n\tat | |
| | | | com.cloud.utils.nio.Task.call(Task.java:83)\n\tat com.cloud.utils.nio.Task.call(Task.java:29)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat | |
| | | | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat | |
| | | | java.base/java.lang.Thread.run(Thread.java:840)\n","lastupdated":"2026-02-11T13:01:46+0100","status":"UNKNOWN","success":false}] | |
+--------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+
```
Expected result: healthcheckfailed=true instead of healthcheckfailed=false
### versions
ACS 4.22.0
### The steps to reproduce the bug
1. ssh into one VR VM
2. shutdown the interfaces e.g. `ifdown eth0 ; ifdown eth1`
3. Wait about 10 minutes
4. Check Alerts in the UI
5. Check `cmk list routers filter=name,hostname,healthchecksfailed,healthcheckresults,state, fetchhealthcheckresults=true listall=true`
### What to do about it?
_No response_
Beitragsleitfaden
Rechercherichtung
Beginne bei VirtualRoutingResource.checkRouterFileSystem und verfolge, wie fehlgeschlagene Health-Check-Ergebnisse durch den KVM-Agenten in die listRouters-API-Antwort gelangen. Reproduziere den Fall mit fetchhealthcheckresults=true und vergleiche healthchecksfailed mit den gemeldeten FAILED-Ergebnissen; abgeschlossen ist die Aufgabe, wenn die API den fehlgeschlagenen Zustand konsistent mit diesen Ergebnissen meldet.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- java, python
- Bereich
- backend, infrastructure
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 35/100