saltstack / saltstack/salt

[BUG] 3003.1 Duplicate Key in vsphere6 module when creating VM with many disks

Open
#60,564 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
When creating a virtual machine using the vsphere6 module, the following error occurs during cloud.create:

A specified parameter was not correct: deviceChange[14].device.key (Cannot add multiple devices using the same device key..)

On subsequent deploys, the index changes to various different devices, or it may intermittently work and deploy a working VM.

Setup
Example cloud.create runner:

salt.runner:
    - name: cloud.create
    - provider: vsphere6
    - instances: maindb3
    - host: esxi-ha-07
    - clonefrom: 2105-rhel7-fips
    - datastore: nvme0
    - num_cpus: 8
    - memory: 128GB
    - folder: /Datacenter/vm/SomeFolder
    - cluster: SomeCluster
    - ssh_username: root
    - devices:  
        scsi: 
          SCSI controller 1:
            type: paravirtual 
          SCSI controller 2:
            type: paravirtual 
          SCSI controller 3:
            type: paravirtual        
        disk:        
          Hard disk 05:
            size: 500
            thin_provision: True
            datastore: nvme0
            controller: SCSI controller 1        
          Hard disk 06:
            size: 500
            thin_provision: True
            datastore: nvme1
            controller: SCSI controller 1        
          Hard disk 07:
            size: 500
            thin_provision: True
            datastore: nvme0
            controller: SCSI controller 1        
          Hard disk 08:
            size: 500
            thin_provision: True
            datastore: nvme1
            controller: SCSI controller 1        
          Hard disk 09:
            size: 500
            thin_provision: True
            datastore: nvme0
            controller: SCSI controller 1        
          Hard disk 10:
            size: 500
            thin_provision: True
            datastore: nvme1
            controller: SCSI controller 1        
          Hard disk 11:
            size: 500
            thin_provision: True
            datastore: nvme0
            controller: SCSI controller 1        
          Hard disk 12:
            size: 500
            thin_provision: True
            datastore: nvme1
            controller: SCSI controller 1        
          Hard disk 13:
            size: 250
            thin_provision: True
            datastore: nvme0
            controller: SCSI controller 2        
          Hard disk 14:
            size: 250
            thin_provision: True
            datastore: nvme1
            controller: SCSI controller 2        
          Hard disk 15:
            size: 250
            thin_provision: True
            datastore: nvme0
            controller: SCSI controller 2        
          Hard disk 16:
            size: 250
            thin_provision: True
            datastore: nvme1
            controller: SCSI controller 2        
          Hard disk 17:
            size: 1000
            thin_provision: True
            datastore: nvme0
            controller: SCSI controller 3        
          Hard disk 18:
            size: 1000
            thin_provision: True
            datastore: nvme1
            controller: SCSI controller 3        
          Hard disk 19:
            size: 1000
            thin_provision: True
            datastore: nvme0
            controller: SCSI controller 3        
          Hard disk 20:
            size: 1000
            thin_provision: True
            datastore: nvme1
            controller: SCSI controller 3
        network:  
          Network adapter 1:
            name: Network
            adapter_type: vmxnet3
            switch_type: standard
            ip: 10.0.0.100
            gateway: 10.0.0.254
            subnet_mask: 255.255.255.0
            connected: True
            start_connected: True   
    - domain: contoso.local
    - extra_config:
        disk.EnableUUID: 'TRUE'
    - dns_servers:
      - 8.8.8.8
      - 8.8.4.4
    - minion:
        id: maindb3
        pillarenv_from_saltenv: True
        use_superseded:
          - module.run

    - script: bootstrap-salt
    - script_args: -x python3 stable 3003.1
    - deploy: True
    - customization: True
    - hardware_version: 15
    - image: rhel7_64Guest
    - template: False
    - power_on: True
    - annotation: Created by Salt-Cloud
    - display_ssh_output: false

Steps to Reproduce the behavior
Run the above salt.runner, or create your own with many disks.

Expected behavior
Virtual machine is created with all disks specified.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3003.1
 
Dependency Versions:
          cffi: 1.12.3
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: 3.1.18
        Jinja2: 2.11.2
       libgit2: 1.1.0
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.19
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: 1.6.1
        Python: 3.6.8 (default, Aug  7 2019, 17:28:10)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 17.0.0
         smmap: 4.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4
 
System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1062.9.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

Additional context
This appears to be a bug in the salt/cloud/clouds/vmware.py module.
In it, instead of using code to create unique device keys for each disk, it uses the following:

def _add_new_hard_disk_helper(
    disk_label,
    size_gb,
    unit_number,
    controller_key=1000,
    thin_provision=False,
    eagerly_scrub=False,
    datastore=None,
    vm_name=None,
):
    random_key = randint(-2099, -2000)

By setting random_key in small int range, the liklihood of colliding IDs is increased.
The code should be rewritten to keep track of ID's generated, and ensure that no two devices are assigned the same ID. This issue could impact ANY other device being added to a virtual machine, but is most likely with disks.

As a work around, I've inceased the random_key int range to :

    random_key = randint(-2999, -2000)

This significantly decreases the liklihood of a collision, but it's still possible. The real fix would be to rewrite the ID generation code to use a 'base ID' for each device type, then increment by 1 for each device added in that type.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/cloud/clouds/vmware.py at _add_new_hard_disk_helper and inspect how device keys are generated for disks and other devices. Reproduce the issue with the provided cloud.create configuration or another VM with many disks. Done means device keys cannot collide and a VM is created with all requested devices.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.