[BUG] salt.states.smartos kvm_reboot has no effect on bhyve branded zones
Open
Nobody has claimed this yet.
bug
severity-medium
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
A bit over a year ago SmartOS got a new brand of zone bhyve which will replace kvm branded in the long run. Both are referred to hvm zones.
Both kvm and bhyve zone brand don't allow modifying some properties unless the zone is stopped, as they are internally running qemu-kvm and bhyve respectively. The smartos.vm_present state has a config option kvm_reboot which will stop, update and start the vm when one of the properties is changed.
This is ignored for bhyve zones, it should not be.
Setup
state 1 (create a bhyve vm)
hypervisor_smartos_vm_bhyvetest:
smartos.vm_present:
- name: bhyvetest
- config:
auto_import: true
kvm_reboot: true
reprovision: True
- vmconfig: {alias: bhyvetest, bootrom: uefi, brand: bhyve, cpu_cap: 200, disks: {disk0: {boot: true,
model: virtio, size: 10240}}, flexible_disk_size: 14592, hostname: bhyvetest, max_physical_memory: 2042, nics: {'00:22:06:03:10:46': {gateways: [10.xx.xx.1],
ips: [10.xx.xx.46/24, '2a02:xx:xx:xx::46/64'], model: virtio, mtu: 9000,
nic_tag: trunk, primary: true, vlan_id: 110}}, quota: 5, resolvers: ['2a02:xx:xx:xx::1', 10.xx.xx.1], tags: {managed: true}, vnc_port: 0, zfs_root_compression: lz4}
state 2 (add a disk while the vm is running)
hypervisor_smartos_vm_bhyvetest:
smartos.vm_present:
- name: bhyvetest
- config:
auto_import: true
kvm_reboot: true
reprovision: True
- vmconfig: {alias: bhyvetest, bootrom: uefi, brand: bhyve, cpu_cap: 200, disks: {disk0: {boot: true,
model: virtio, size: 10240}, disk1: {boot: false, model: virtio,
size: 4096}}, flexible_disk_size: 14592, hostname: bhyvetest, max_physical_memory: 2042, nics: {'00:22:06:03:10:46': {gateways: [10.xx.xx.1],
ips: [10.xx.xx.46/24, '2a02:xx:xx:xx::46/64'], model: virtio, mtu: 9000,
nic_tag: trunk, primary: true, vlan_id: 110}}, quota: 5, resolvers: ['2a02:xx:xx:xx::1', 10.xx.xx.1], tags: {managed: true}, vnc_port: 0, zfs_root_compression: lz4}
Steps to Reproduce the behavior
- run state 1 to create the hvm
- run state 2 to try and add disk
Results in
ID: hypervisor_smartos_vm_bhyvetest
Function: smartos.vm_present
Name: bhyvetest
Result: False
Comment: updates to disks are only allowed when state is "stopped", currently: running (running)
Started: 10:54:06.576278
Duration: 3497.895 ms
Changes:
Which is correct, the module was written before bhyve branded zones existed. The error comes direct from the stdout of the underlying tool that is called.
You can see this in the debug output
debug output
[INFO ] Running state [bhyvetest] at time 10:54:06.576279
[INFO ] Executing state smartos.vm_present for [bhyvetest]
[DEBUG ] smartos.vm_present::bhyvetest::config - {'kvm_reboot': True, 'auto_import': True, 'auto_lx_vars': True, 'reprovision': True, 'enforce_tags': True, 'enforce_routes': True, 'enforce_internal_metadata': True, 'enforce_customer_metadata': True}
[DEBUG ] smartos.vm_present::bhyvetest::vmconfig - OrderedDict([('alias', 'bhyvetest'), ('bootrom', 'uefi'), ('brand', 'bhyve'), ('cpu_cap', 200), ('disks', [OrderedDict([('boot', True), ('model', 'virtio'), ('size', 10240), ('path', 'disk0')]), OrderedDict([('boot', False), ('media', 'cdrom'), ('model', 'ahci'), ('size', 4096), ('path', 'disk1')])]), ('flexible_disk_size', 14592), ('hostname', 'bhyvetest'), ('max_physical_memory', 2042), ('nics', [OrderedDict([('gateways', ['10.xx.xx.1']), ('ips', ['10.xx.xx.46/24', '2a02:xx:xx:xx::46/64']), ('model', 'virtio'), ('mtu', 9000), ('nic_tag', 'trunk'), ('primary', True), ('vlan_id', 110), ('mac', '00:22:06:03:10:46')])]), ('quota', 5), ('resolvers', ['2a02:xx:xx:xx::1', '10.xx.xx.1']), ('tags', OrderedDict([('managed', True)])), ('vnc_port', 0), ('zfs_root_compression', 'lz4')])
[INFO ] Executing command 'vmadm' in directory '/root'
[DEBUG ] stdout: smtest
bhyvetest
[INFO ] Executing command 'vmadm' in directory '/root'
[DEBUG ] stdout: 219a9a72-9072-43f9-d706-c375afa93b7f
[INFO ] Executing command 'vmadm' in directory '/root'
[DEBUG ] stdout: {
"zonename": "219a9a72-9072-43f9-d706-c375afa93b7f",
"autoboot": true,
"brand": "bhyve",
"limit_priv": "default,-file_link_any,-net_access,-proc_fork,-proc_info,-proc_session",
"v": 1,
"create_timestamp": "2021-02-27T10:50:31.542Z",
"cpu_shares": 100,
"max_lwps": 2000,
"max_msg_ids": 4096,
"max_sem_ids": 4096,
"max_shm_ids": 4096,
"max_shm_memory": 2042,
"zfs_io_priority": 100,
"max_physical_memory": 2042,
"max_locked_memory": 2042,
"max_swap": 2042,
"cpu_cap": 200,
"billing_id": "00000000-0000-0000-0000-000000000000",
"owner_uuid": "00000000-0000-0000-0000-000000000000",
"hostname": "bhyvetest",
"resolvers": [
"2a02:xx:xx:xx::1",
"10.xx.xx.1"
],
"alias": "bhyvetest",
"ram": 256,
"com1": "/dev/zconsole",
"com2": "socket,/tmp/vm.ttyb",
"bootrom": "uefi",
"zlog_mode": "g--",
"zlog_name": "platform.log",
"vcpus": 1,
"flexible_disk_size": 10496,
"device": null,
"disks": [
{
"path": "/dev/zvol/rdsk/zones/219a9a72-9072-43f9-d706-c375afa93b7f/disk0",
"boot": true,
"model": "virtio",
"media": "disk",
"pci_slot": "0:4:0",
"uuid": "b49a2e09-1159-c35a-c6a4-a049d6f648f0",
"zfs_filesystem": "zones/219a9a72-9072-43f9-d706-c375afa93b7f/disk0",
"zpool": "zones",
"size": 10240,
"compression": "lz4",
"refreservation": 10563,
"block_size": 8192
}
],
"nics": [
{
"interface": "net0",
"mac": "00:22:06:03:10:46",
"vlan_id": 110,
"nic_tag": "trunk",
"gateway": "10.xx.xx.1",
"gateways": [
"10.xx.xx.1"
],
"netmask": "255.255.255.0",
"ip": "10.xx.xx.46",
"ips": [
"10.xx.xx.46/24",
"2a02:xx:xx:xx::46/64"
],
"model": "virtio",
"mtu": 9000,
"primary": true
}
],
"uuid": "219a9a72-9072-43f9-d706-c375afa93b7f",
"zone_state": "running",
"zonepath": "/zones/219a9a72-9072-43f9-d706-c375afa93b7f",
"hvm": true,
"zoneid": 13,
"zonedid": 105,
"last_modified": "2021-02-27T10:50:37.000Z",
"firewall_enabled": false,
"server_uuid": "00000000-0000-0000-0000-3cecef0c783a",
"platform_buildstamp": "20210215T170311Z",
"state": "running",
"boot_timestamp": "2021-02-27T10:50:36.000Z",
"init_restarts": 0,
"pid": 684585,
"customer_metadata": {},
"internal_metadata": {},
"routes": {},
"tags": {
"managed": true
},
"quota": 5,
"zfs_root_compression": "lz4",
"zfs_root_recsize": 131072,
"zfs_filesystem": "zones/219a9a72-9072-43f9-d706-c375afa93b7f",
"zpool": "zones",
"snapshots": [],
"free_space": 256
}
[DEBUG ] smartos.vm_present::enforce_tags = True
[DEBUG ] smartos.vm_present::enforce_customer_metadata = True
[DEBUG ] smartos.vm_present::enforce_internal_metadata = True
[DEBUG ] smartos.vm_present::enforce_routes = True
[INFO ] Executing command 'vmadm' in directory '/root'
[DEBUG ] stdout: 219a9a72-9072-43f9-d706-c375afa93b7f
[DEBUG ] LazyLoaded temp.file
[INFO ] Executing command 'vmadm' in directory '/root'
[DEBUG ] stdout: 219a9a72-9072-43f9-d706-c375afa93b7f
[INFO ] Executing command 'vmadm' in directory '/root'
[DEBUG ] stdout: {
"zonename": "219a9a72-9072-43f9-d706-c375afa93b7f",
"autoboot": true,
"brand": "bhyve",
"limit_priv": "default,-file_link_any,-net_access,-proc_fork,-proc_info,-proc_session",
"v": 1,
"create_timestamp": "2021-02-27T10:50:31.542Z",
"cpu_shares": 100,
"max_lwps": 2000,
"max_msg_ids": 4096,
"max_sem_ids": 4096,
"max_shm_ids": 4096,
"max_shm_memory": 2042,
"zfs_io_priority": 100,
"max_physical_memory": 2042,
"max_locked_memory": 2042,
"max_swap": 2042,
"cpu_cap": 200,
"billing_id": "00000000-0000-0000-0000-000000000000",
"owner_uuid": "00000000-0000-0000-0000-000000000000",
"hostname": "bhyvetest",
"resolvers": [
"2a02:xx:xx:xx::1",
"10.xx.xx.1"
],
"alias": "bhyvetest",
"ram": 256,
"com1": "/dev/zconsole",
"com2": "socket,/tmp/vm.ttyb",
"bootrom": "uefi",
"zlog_mode": "g--",
"zlog_name": "platform.log",
"vcpus": 1,
"flexible_disk_size": 10496,
"device": null,
"disks": [
{
"path": "/dev/zvol/rdsk/zones/219a9a72-9072-43f9-d706-c375afa93b7f/disk0",
"boot": true,
"model": "virtio",
"media": "disk",
"pci_slot": "0:4:0",
"uuid": "b49a2e09-1159-c35a-c6a4-a049d6f648f0",
"zfs_filesystem": "zones/219a9a72-9072-43f9-d706-c375afa93b7f/disk0",
"zpool": "zones",
"size": 10240,
"compression": "lz4",
"refreservation": 10563,
"block_size": 8192
}
],
"nics": [
{
"interface": "net0",
"mac": "00:22:06:03:10:46",
"vlan_id": 110,
"nic_tag": "trunk",
"gateway": "10.xx.xx.1",
"gateways": [
"10.xx.xx.1"
],
"netmask": "255.255.255.0",
"ip": "10.xx.xx.46",
"ips": [
"10.xx.xx.46/24",
"2a02:xx:xx:xx::46/64"
],
"model": "virtio",
"mtu": 9000,
"primary": true
}
],
"uuid": "219a9a72-9072-43f9-d706-c375afa93b7f",
"zone_state": "running",
"zonepath": "/zones/219a9a72-9072-43f9-d706-c375afa93b7f",
"hvm": true,
"zoneid": 13,
"zonedid": 105,
"last_modified": "2021-02-27T10:50:37.000Z",
"firewall_enabled": false,
"server_uuid": "00000000-0000-0000-0000-3cecef0c783a",
"platform_buildstamp": "20210215T170311Z",
"state": "running",
"boot_timestamp": "2021-02-27T10:50:36.000Z",
"init_restarts": 0,
"pid": 684585,
"customer_metadata": {},
"internal_metadata": {},
"routes": {},
"tags": {
"managed": true
},
"quota": 5,
"zfs_root_compression": "lz4",
"zfs_root_recsize": 131072,
"zfs_filesystem": "zones/219a9a72-9072-43f9-d706-c375afa93b7f",
"zpool": "zones",
"snapshots": [],
"free_space": 256
}
[INFO ] Executing command 'vmadm' in directory '/root'
[DEBUG ] stderr: VALID 'update' payload for bhyve brand VMs.
[INFO ] Executing command 'vmadm' in directory '/root'
[ERROR ] Command 'vmadm' failed with return code: 1
[ERROR ] stderr: updates to disks are only allowed when state is "stopped", currently: running (running)
[ERROR ] retcode: 1
[ERROR ] updates to disks are only allowed when state is "stopped", currently: running (running)
[INFO ] Completed state [bhyvetest] at time 10:54:10.074173 (duration_in_ms=3497.895)
Expected behavior
kvm_reboot to apply to both kvm and bhyve branded zones.
Screenshots
n/a
Versions Report
salt --versions-report
Salt Version: Salt: 3002.5Dependency Versions:
cffi: 1.14.4
cherrypy: Not Installed
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.2
libgit2: Not Installed
M2Crypto: Not Installed
Mako: 1.1.4
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: Not Installed
pycryptodome: 3.9.9
pygit2: Not Installed
Python: 3.8.7 (default, Jan 30 2021, 13:54:42)
python-gnupg: 0.4.6
PyYAML: 5.4.1
PyZMQ: 19.0.2
smmap: 4.0.0
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.2
System Versions:
dist: sunos 5.11
locale: utf-8
machine: i86pc
release: 5.11
system: SunOS
version: SunOS 5.11
Additional context
Ideally we rename kvm_reboot to hvm_reboot and keep kvm_reboot around for backwards compat. But a simple fix for now is:
--- /opt/salt/lib/python3.7/site-packages/salt/states/smartos.py 2021-02-11 17:35:31.204041643 +0000
+++ smartos.py.hack 2021-02-27 11:40:09.111092840 +0000
@@ -1092,7 +1092,7 @@
kvm_needs_start = False
if not __opts__["test"] and vmconfig["changed"]:
# stop kvm if disk updates and kvm_reboot
- if vmconfig["current"]["brand"] == "kvm" and config["kvm_reboot"]:
+ if vmconfig["current"]["brand"] in ["bhyve", "kvm"] and config["kvm_reboot"]:
if (
"add_disks" in vmconfig["changed"]
or "update_disks" in vmconfig["changed"]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the salt.states.smartos.vm_present implementation and its kvm_reboot handling. Reproduce the issue with the two provided bhyve configurations, then verify that changing a disk while the VM is running stops and restarts the zone when kvm_reboot is enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100