apache / apache/cloudstack

VR healthcheck wrong state

Open
#12,630 0 comments 0 reactions 0 assignees View on GitHub
component:healthcheck
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

### problem

The listRouters API-call always returns healthcheckfailed false, even if the healthcheck fails.

Alert in CloudStack UI:

```
Health checks failed: 1 failing checks on router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16
```

Error in EventQueue

```
{"eventDateTime":"2026-02-09 14:01:22 +0100","subject":"Health checks failed: 1 failing checks on router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16","dataCenterId":"a8f95837-c05c-4a97-b100-53f87de8df24","event":"ALERT.SERVICE.DOMAINROUTER","body":"Health checks failed: 1 failing checks on router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16","podId":"090ccb90-9411-4bab-bb0c-f5ab61e34ebb"}
{"eventDateTime":"2026-02-09 14:01:22 +0100","entityuuid":"c0c6a15f-3ad8-4936-ba6e-abe802ddba16","description":"Router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16 has failing checks: router_version_check.py","event":"ROUTER.HEALTH.CHECKS","user":"c302beae-cb9d-11f0-a957-fa163ed04c4b","account":"c302a9b2-cb9d-11f0-a957-fa163ed04c4b","entity":"DomainRouter","status":"Completed"}
```

API call via cmk

```
(cloudstackmsu) 🐄 > list routers name=r-8-VM filter=name,hostname,healthchecksfailed,healthcheckresults,state, fetchhealthcheckresults=true listall=true
router:
count = 1
+--------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+
| NAME | HOSTNAME | HEALTHCHECKSFAILED | HEALTHCHECKRESULTS | STATE |
+--------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+
| r-8-VM | srv1-214 | false | [{"checkname":"connectivity.test","checktype":"basic","details":"Failed to fetch results with details: java.io.IOException: Stream closed\n\tat | Running |
| | | | java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)\n\tat java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)\n\tat | |
| | | | java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)\n\tat java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)\n\tat | |
| | | | java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)\n\tat java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)\n\tat | |
| | | | java.base/java.io.BufferedReader.fill(BufferedReader.java:162)\n\tat java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)\n\tat | |
| | | | java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)\n\tat com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)\n\tat | |
| | | | com.cloud.utils.script.Script.execute(Script.java:336)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:632)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:620)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:431)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:403)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:239)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:130)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)\n\tat | |
| | | | com.cloud.agent.Agent.processRequest(Agent.java:813)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)\n\tat | |
| | | | com.cloud.utils.nio.Task.call(Task.java:83)\n\tat com.cloud.utils.nio.Task.call(Task.java:29)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat | |
| | | | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat | |
| | | | java.base/java.lang.Thread.run(Thread.java:840)\n","lastupdated":"2026-02-11T13:01:46+0100","status":"FAILED","success":false},{"checkname":"filesystem.writable.test","checktype":"basic","details":"Failed | |
| | | | to fetch results with details: java.io.IOException: Stream closed\n\tat java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)\n\tat | |
| | | | java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)\n\tat java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)\n\tat | |
| | | | java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)\n\tat java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)\n\tat | |
| | | | java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)\n\tat java.base/java.io.BufferedReader.fill(BufferedReader.java:162)\n\tat | |
| | | | java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)\n\tat java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)\n\tat | |
| | | | com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)\n\tat com.cloud.utils.script.Script.execute(Script.java:336)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:632)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:620)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:431)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:403)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:239)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:130)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)\n\tat | |
| | | | com.cloud.agent.Agent.processRequest(Agent.java:813)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)\n\tat | |
| | | | com.cloud.utils.nio.Task.call(Task.java:83)\n\tat com.cloud.utils.nio.Task.call(Task.java:29)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat | |
| | | | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat | |
| | | | java.base/java.lang.Thread.run(Thread.java:840)\n","lastupdated":"2026-02-11T13:01:46+0100","status":"UNKNOWN","success":false}] | |
+--------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+
```

Expected result: healthcheckfailed=true instead of healthcheckfailed=false

### versions

ACS 4.22.0

### The steps to reproduce the bug

1. ssh into one VR VM
2. shutdown the interfaces e.g. `ifdown eth0 ; ifdown eth1`
3. Wait about 10 minutes
4. Check Alerts in the UI
5. Check `cmk list routers filter=name,hostname,healthchecksfailed,healthcheckresults,state, fetchhealthcheckresults=true listall=true`

### What to do about it?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at VirtualRoutingResource.checkRouterFileSystem and trace how failed health-check results move through the KVM agent into the listRouters API response. Reproduce the case using fetchhealthcheckresults=true and compare healthchecksfailed with the reported FAILED results; done means the API reports the failing state consistently with those results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.