apache / apache/cloudstack

KVM NAS backup: hard NFS mount default causes VM outages when backup storage is unreachable

オープン
#12,829 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
component:backup no-issue-activity
主要言語
Java
スター
3.1k
フォーク
1.4k
平均マージ
6日 19時間
マージ済み PR(30日)
32

説明

## Description

When a NAS backup repository becomes temporarily unreachable, running VMs on the KVM host can be affected — including VMs that have no relationship to the backup operation. This is because the NFS mount used by `nasbackup.sh` defaults to `hard` mode, which blocks indefinitely when the NFS server is unresponsive.

## Root Cause

The `backup_repository.mount_opts` column defaults to empty. When `nasbackup.sh` calls `mount_operation()`, it mounts the backup NFS share with no special options, which defaults to NFS `hard` mode:

```bash
# nasbackup.sh mount_operation()
mount -t ${NAS_TYPE} ${NAS_ADDRESS} ${mount_point} $([[ ! -z "${MOUNT_OPTS}" ]] && echo -o ${MOUNT_OPTS})
```

With `hard` mode, any I/O operation on the NFS mount blocks indefinitely when the server is unreachable. This causes a cascade:

1. `nasbackup.sh` hangs on NFS I/O (write, sync, or umount)
2. The CloudStack agent is blocked because `nasbackup.sh` runs as a child process of the agent JVM
3. The blocked agent cannot process **any** VM operations (PlugNic, Stop, Migrate, etc.) — all commands queue behind the stuck backup
4. On the host kernel level, NFS `hard` mount stalls can cause I/O waits that affect all processes, including QEMU instances for unrelated VMs
5. VMs experience I/O timeouts — Windows guests BSOD with `KERNEL_DATA_INPAGE_ERROR`, Linux guests may freeze

## Evidence (from production CloudStack 4.20)

NFS server 172.16.3.63 experienced intermittent connectivity issues. Host dmesg showed repeated cycles:
```
nfs: server 172.16.3.63 not responding, still trying
nfs: server 172.16.3.63 OK
nfs: server 172.16.3.63 not responding, still trying
nfs: server 172.16.3.63 OK
```

Impact:
- A Windows VM (`citytravelsacco`, i-2-1651-VM) on the same host **crashed with BSOD** (`KERNEL_DATA_INPAGE_ERROR`) even though its disk is on **local storage**, not NFS
- The CloudStack agent was blocked for 3+ hours by a stuck `nasbackup.sh` process, preventing all VM management operations on the host
- A NIC hot-plug operation queued for 30+ minutes waiting for the agent to become responsive

## Suggested Fix

1. **Default `mount_opts` for NAS backup repositories to `soft,timeo=50,retrans=3`** — this causes NFS operations to fail after ~15 seconds instead of blocking forever. A failed backup is far preferable to crashing production VMs.

2. **Add a timeout wrapper to `nasbackup.sh`** — if the entire backup operation exceeds a configurable duration, kill it cleanly (resume paused VM, unmount, exit with error).

3. **Document the risk** — warn administrators that empty `mount_opts` on NAS backup repositories defaults to NFS `hard` mode, which can cause host-wide I/O stalls.

Note: `soft` mount may cause backup data corruption if the NFS server recovers mid-write, but this only affects the backup copy — not the production VM. A corrupted backup can be retried; a crashed production VM cannot be un-crashed.

## Workaround

Manually set mount options on the backup repository:
```sql
UPDATE cloud.backup_repository SET mount_opts='soft,timeo=50,retrans=3' WHERE id=;
```

And update `/etc/fstab` on KVM hosts if the NFS backup share is persistently mounted:
```
172.16.3.63:/ACS /tmp/nasbackup nfs soft,timeo=50,retrans=3,_netdev 0 0
```

## Versions

- CloudStack 4.20
- NFS v4.1
- KVM hosts: Debian/Ubuntu with kernel 5.x/6.x

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

nasbackup.sh とそのエントリーポイント mount_operation() から始め、次に backup_repository.mount_opts のデフォルト値と CloudStack エージェントのプロセス境界を追跡します。Issue に記載されている到達不能な NFS の動作を再現または確認します。完了には、hard mount に対する合意済みの緩和策、必要な timeout またはクリーンアップ動作、および運用上のリスクの文書化が必要です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java, linux, shell
領域
backend, infrastructure, operating-systems
issue の種類
バグ
難易度
5/5
見積もり時間
1週間以上
活発さ
静か
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。