saltstack / saltstack/salt

salt-cloud fails during fixed IPs creation and provisioning GCE instances

Open
#56,823 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Core Salt-Cloud severity-high
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
salt-cloud 2019.2.3, gce provider.
salt-cloud fails during fixed IPs creation and provisioning GCE instances.

Setup

/etc/salt/cloud.profiles.d/gce.conf:
vpn-profile:
  image: debian-9
  ssh_username: gceuser
  ssh_keyfile: /root/.ssh/id_rsa_gce
  size: f1-micro
  network: default
  subnetwork: default
  tags: '["allow-all"]'
  metadata: '{"sshKeys": "MYKEY"}'
  use_persistent_disk: True
  delete_boot_pd: True
  ssh_interface: public_ips
  ip_forwarding: False
  deploy: True
  make_master: False
  provider: MYPROVIDER

/etc/salt/cloud.maps.d/vpn.conf:
vpn-profile:
  - vpn1:
      location: europe-west4-a
      external_ip: "vpn1"
  - vpn2:
      location: europe-west4-b
      external_ip: "vpn2"
  - vpn3:
      location: europe-west4-c
      external_ip: "vpn3"

According to documentation https://docs.saltstack.com/en/latest/topics/cloud/gce.html#external-ip
"vpnX" static (fixed) IPs should be created:

Optionally, pass the name of a GCE address to use a fixed IP address. If the address does not already exist, it will be created.

Steps to Reproduce the behavior
Run setup:

/etc/salt/cloud.providers.d# salt-cloud -m /etc/salt/cloud.maps.d/vpn.conf -P
The following virtual machines are set to be created:
  vpn2
  vpn1
  vpn3

Proceed? [N/y] y
... proceeding
[ERROR   ] Caught Exception, terminating workers
TRACE: can't serialize <GCERegion id="1340" name="europe-west4", status="UP">
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/gce.py", line 511, in __create_orget_address
    addy = conn.ex_get_address(name, region)
  File "/usr/lib/python3/dist-packages/libcloud/compute/drivers/gce.py", line 6556, in ex_get_address
    response = self.connection.request(request, method='GET').object
  File "/usr/lib/python3/dist-packages/libcloud/compute/drivers/gce.py", line 120, in request
    response = super(GCEConnection, self).request(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/libcloud/common/google.py", line 806, in request
    *args, **kwargs)
  File "/usr/lib/python3/dist-packages/libcloud/common/base.py", line 871, in request
    response = responseCls(**kwargs)
  File "/usr/lib/python3/dist-packages/libcloud/common/base.py", line 182, in __init__
    self.object = self.parse_body()
  File "/usr/lib/python3/dist-packages/libcloud/common/google.py", line 278, in parse_body
    raise ResourceNotFoundError(message, self.status, code)
libcloud.common.google.ResourceNotFoundError: {'domain': 'global', 'reason': 'notFound', 'message': "The resource 'projects/blah/regions/europe-west4/addresses/vpn2' was not found"}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 1991, in fire_event
    event.fire_event(msg, tag)
  File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 708, in fire_event
    'Dict object expected, not \'{0}\'.'.format(data)
ValueError: Dict object expected, not 'create address'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/cloud/__init__.py", line 67, in _call
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/cloud/__init__.py", line 2294, in create_multiprocessing
    local_master=parallel_data['local_master']
  File "/usr/lib/python3/dist-packages/salt/cloud/__init__.py", line 1253, in create
    output = self.clouds[func](vm_)
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/gce.py", line 2579, in create
    node_info = request_instance(vm_)
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/gce.py", line 2485, in request_instance
    external_ip = __create_orget_address(conn, external_ip, region)
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/gce.py", line 517, in __create_orget_address
    new_addy = create_address(addr_kwargs, "function")
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/gce.py", line 1263, in create_address
    transport=__opts__['transport']
  File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 1998, in fire_event
    event.fire_event(args, tag)
  File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 729, in fire_event
    dump_data = self.serial.dumps(data, use_bin_type=True)
  File "/usr/lib/python3/dist-packages/salt/payload.py", line 240, in dumps
    return msgpack.dumps(msg, default=ext_type_encoder, use_bin_type=use_bin_type)
  File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)
  File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)
  File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)
  File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)
TypeError: can't serialize <GCERegion id="1340" name="europe-west4", status="UP">

Error: There was a query error: Exception caught
Caught Exception, terminating workers
TRACE: can't serialize <GCERegion id="1340" name="europe-west4", status="UP">
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/gce.py", line 511, in __create_orget_address
    addy = conn.ex_get_address(name, region)
  File "/usr/lib/python3/dist-packages/libcloud/compute/drivers/gce.py", line 6556, in ex_get_address
    response = self.connection.request(request, method='GET').object
  File "/usr/lib/python3/dist-packages/libcloud/compute/drivers/gce.py", line 120, in request
    response = super(GCEConnection, self).request(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/libcloud/common/google.py", line 806, in request
    *args, **kwargs)
  File "/usr/lib/python3/dist-packages/libcloud/common/base.py", line 871, in request
    response = responseCls(**kwargs)
  File "/usr/lib/python3/dist-packages/libcloud/common/base.py", line 182, in __init__
    self.object = self.parse_body()
  File "/usr/lib/python3/dist-packages/libcloud/common/google.py", line 278, in parse_body
    raise ResourceNotFoundError(message, self.status, code)
libcloud.common.google.ResourceNotFoundError: {'domain': 'global', 'reason': 'notFound', 'message': "The resource 'projects/blah/regions/europe-west4/addresses/vpn2' was not found"}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 1991, in fire_event
    event.fire_event(msg, tag)
  File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 708, in fire_event
    'Dict object expected, not \'{0}\'.'.format(data)
ValueError: Dict object expected, not 'create address'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/cloud/__init__.py", line 67, in _call
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/cloud/__init__.py", line 2294, in create_multiprocessing
    local_master=parallel_data['local_master']
  File "/usr/lib/python3/dist-packages/salt/cloud/__init__.py", line 1253, in create
    output = self.clouds[func](vm_)
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/gce.py", line 2579, in create
    node_info = request_instance(vm_)
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/gce.py", line 2485, in request_instance
    external_ip = __create_orget_address(conn, external_ip, region)
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/gce.py", line 517, in __create_orget_address
    new_addy = create_address(addr_kwargs, "function")
  File "/usr/lib/python3/dist-packages/salt/cloud/clouds/gce.py", line 1263, in create_address
    transport=__opts__['transport']
  File "/usr/lib/python3/dist-packages/salt/utils/cloud.py", line 1998, in fire_event
    event.fire_event(args, tag)
  File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 729, in fire_event
    dump_data = self.serial.dumps(data, use_bin_type=True)
  File "/usr/lib/python3/dist-packages/salt/payload.py", line 240, in dumps
    return msgpack.dumps(msg, default=ext_type_encoder, use_bin_type=use_bin_type)
  File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)
  File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)
  File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)
  File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)
TypeError: can't serialize <GCERegion id="1340" name="europe-west4", status="UP">

So, neither fixed IPs nor GCE instances were created.

Expected behavior
"vpnX" fixed IPs and GCE instances should be created, fixed IPs should be mapped to GCE instances, salt-minion and configuration are applied.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
Salt Version:
           Salt: 2019.2.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.5.0
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
   python-gnupg: 0.3.9
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.12 
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-12-amd64
         system: Linux
        version: debian 9.12

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/cloud/clouds/gce.py at __create_orget_address and create_address, then trace event handling in salt/utils/cloud.py. Reproduce the missing-address path with the supplied GCE profile and map, and verify that fixed addresses and GCE instances are created, mapped, and provisioned without the serialization error.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.