[BUG] Creating VMs with a ZFS pool is racy

Description

I'm using virt.running to create a VM, backed by a libvirt ZFS pool. When I run the state, I get an error like salt.exceptions.CommandExecutionError: Unable to open stream for '/dev/zvol/rpool/vms/foo_main': No such file or directory. After the state fails, that path does seem to be present.

I tried patching _volume_upload to run os.system('ls -lR /dev/zd* /dev/zvol/') after the volume should be created but before the upload begins. The zd0 block device was present, but not the symlink under zvol. I added a sleep followed by another ls call, and the state began working and ls reported the symlink as present after the sleep.

This looks like there's some sort of race between the volume getting fully set up and uploading the image to it. Unfortunately, I have no idea if libvirt provides any mechanism for doing volume creation more synchronously. (Another workaround might be to check if the path exists and not continue until it does.)

Setup

I tried this with Salt 3004.1 and 3006.1.

State looks like:

{% set vm_name = "foo" %}
VM create {{vm_name}}:
  virt.running:
    - name: {{vm_name}}   
    - seed: false
    - mem: 512
    - disks:
      - name: main
        size: 10240
        pool: zfs-vms
        format: raw
        image: {{vm_image}}.raw

The libvirt pool is ZFS-backed (although without my change in #64279 I can't use Salt to create it). I somewhat doubt this is entirely ZFS-specific, though it would require something where the volume is relatively slow to create, so it certainly could be more common on some pool types than others.

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
Use the state above, setting up a ZFS-backed libvirt pool. Run state.apply.

Observe error:

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/virt.py", line 8945, in _volume_upload
    vol_obj.upload(
  File "/opt/saltstack/salt/extras-3.10/libvirt.py", line 4203, in upload
    raise libvirtError('virStorageVolUpload() failed')
libvirt.libvirtError: Unable to open stream for '/dev/zvol/rpool/vms/foo_main': No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2385, in call
    ret = self.states[cdata["full"]](
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1232, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1247, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1280, in wrapper
    return f(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/virt.py", line 1009, in running
    ret = defined(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/virt.py", line 690, in defined
    __salt__["virt.init"](
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1232, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1247, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/virt.py", line 2860, in init
    _disk_volume_create(conn, _disk, seeder if seed else None, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/virt.py", line 1587, in _disk_volume_create
    _volume_upload(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/virt.py", line 8957, in _volume_upload
    raise CommandExecutionError(err.get_error_message())
salt.exceptions.CommandExecutionError: Unable to open stream for '/dev/zvol/rpool/vms/foo_main': No such file or directory

Expected behavior
The new volume should be created and the image should be uploaded to it.

Screenshots
N/A

Versions Report

Salt Version:
          Salt: 3004.1
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.3
       libgit2: Not Installed
      M2Crypto: 0.38.0
          Mako: Not Installed
       msgpack: 1.0.3
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.11.0
        pygit2: Not Installed
        Python: 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0]
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 22.3.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 22.04 Jammy Jellyfish
        locale: utf-8
       machine: x86_64
       release: 5.15.0-71-generic
        system: Linux
       version: Ubuntu 22.04 Jammy Jellyfish

Salt Version:
          Salt: 3006.1
 
Python Version:
        Python: 3.10.11 (main, May  5 2023, 02:31:54) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
        relenv: 0.12.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 22.04.2 jammy
        locale: iso8859-1
       machine: x86_64
       release: 5.15.0-71-generic
        system: Linux
       version: Ubuntu 22.04.2 jammy

Additional context
Add any other context about the problem here.

Start in salt/modules/virt.py at _disk_volume_create and _volume_upload, following the call from salt/states/virt.py running and defined. Reproduce with the provided state.apply example and a ZFS-backed libvirt pool, then verify that volume creation completes before image upload and that the expected volume path is available without the reported error.