apache / apache/cloudstack

KVM: kvmheartbeat.sh / kvmspheartbeat.sh hardcoded sysrq reboot causes false-positive host fencing on LINSTOR/DRBD primary storage

未关闭
#13,089 1 条评论 0 个 reaction 已指派 1 人 已被 @jmsperu 认领 在 GitHub 查看
component:kvm type:enhancement
主要语言
Java
星标
3.1k
派生
1.4k
平均合并
6 天 19 小时
30 天内合并 PR
32

描述

### Description

The KVM agent's storage heartbeat scripts (`kvmheartbeat.sh`, `kvmspheartbeat.sh`) hard-code an immediate kernel-level reboot via `echo b > /proc/sysrq-trigger` when a heartbeat write to primary storage times out. This:

1. Bypasses all OS-level shutdown protections (no clean filesystem unmount, no graceful VM stop)
2. Drops ALL running VMs on the host instantly
3. Triggers HA cascade — surviving hosts get flooded with VM restart requests
4. Cannot be replaced with a gentler action (restart agent, graceful reboot, log-only) without forking the script

### Affected files

- `scripts/vm/hypervisor/kvm/kvmheartbeat.sh` (around line 162)
- `scripts/vm/hypervisor/kvm/kvmspheartbeat.sh` (around line 64)

```sh
# Both scripts contain:
/usr/bin/logger -t heartbeat "...will reboot system..."
sync &
sleep 5
echo b > /proc/sysrq-trigger # <-- hardcoded panic-reboot
```

### Reproduction

1. CloudStack 4.22.0.0 KVM hypervisor running Ubuntu 24.04
2. Primary storage: **LINSTOR with DRBD** replication (Linstor storage pool)
3. Trigger parallel DRBD resyncs across many resources (e.g. after a peer-node reboot, or `linstor resource resume-sync` after a maintenance pause)
4. While resync I/O contends with normal I/O on the same disks, the heartbeat write to its `hb` file occasionally takes longer than the hardcoded timeout
5. The host immediately force-reboots itself via sysrq, even though no actual fault exists

### Real-world impact

We hit this 5+ times within 4 hours during recovery from an unrelated incident. Cascade pattern:

- Heartbeat times out on host A → sysrq reboot → 90+ VMs dropped
- HA worker reschedules those VMs onto host B
- Host B's I/O spikes → its heartbeat times out → sysrq reboot → all its VMs dropped
- Host A comes back, repeat

Each reboot took ~3 minutes; total customer-visible outage was several hours. Patching the script line out (`echo b > /proc/sysrq-trigger` → log-only) immediately stopped the cascade.

The exact log line preceding each reboot:

```
heartbeat[68685]: kvmspheartbeat.sh will reboot system because it was unable to write the heartbeat to the storage.
```

### Affected versions

- CloudStack: **4.22.0.0** (cloudstack-agent / cloudstack-common 4.22.0.0)
- Hypervisor: Ubuntu 24.04 LTS / KVM, libvirt 10.x
- Storage: LINSTOR 1.33.2 / DRBD 9.3.1 (LINBIT public PPA)
- Reproduces on both single-cluster and multi-cluster zones

### Why this matters

The assumption "heartbeat write timeout = host is dead" was reasonable for **NFS shared storage** where transient I/O latency is rare and the only failure mode of concern is split-brain during a real network partition.

With **LINSTOR/DRBD** (or any local storage doing replication), the same disk serves application I/O, replication I/O, and heartbeat I/O — heartbeat can be transiently delayed without the host being dead. A fence-on-failure mechanism shouldn't be a binary panic-button: it should be **configurable**, and ideally **graceful** by default.

### Existing partial workaround (poorly documented)

There is already an agent property `reboot.host.and.alert.management.on.heartbeat.timeout` (default `true`) that, when set to `false`, prevents the Java-side `KVMHAMonitor` from invoking the shell script in fence mode (cflag=1). Setting:

```properties
reboot.host.and.alert.management.on.heartbeat.timeout=false
```

avoids the reboot entirely. This is an undocumented but effective workaround for the described issue.

However, this is a binary on/off — it doesn't support intermediate fence actions (restart agent, graceful reboot, log-with-alert). Operators want a middle ground: detect the failure, *react* in a less destructive way.

### Proposed enhancement

Add a finer-grained `kvm.heartbeat.fence.action` property that **supersedes** the binary boolean when set:

```properties
# Action when storage heartbeat write fails persistently
# Values: reboot | graceful-reboot | restart-agent | log-only
# Default: reboot (preserves current behavior for backward compatibility)
kvm.heartbeat.fence.action=graceful-reboot
```

Action semantics:

- `reboot` — current behavior (sysrq-trigger), kept as fallback for backward compat
- `graceful-reboot` — `systemctl reboot` instead of sysrq, lets VMs stop cleanly
- `restart-agent` — restart `cloudstack-agent` only; running VMs survive
- `log-only` — log + alert, take no automatic action (admin investigates)

Existing `reboot.host.and.alert.management.on.heartbeat.timeout=false` continues to work as a complete bypass (Java-side, never invokes the shell script in fence mode).

### Workarounds available today

1. **Set `reboot.host.and.alert.management.on.heartbeat.timeout=false`** in `/etc/cloudstack/agent/agent.properties` (Java-side flag — best official option until a proper fix lands)
2. **Patch both shell scripts** to replace `echo b > /proc/sysrq-trigger` with a `logger` line — survives until next package upgrade
3. **Move heartbeat target to host-local storage** that's not under DRBD I/O contention

### Willingness to contribute

A follow-up PR implementing the configurable `kvm.heartbeat.fence.action` design above is in progress.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。