saltstack / saltstack/salt

[salt-cloud]Defining `script_args: -j json_object` at provider level does not work with openstack driver

Open
#52,983 19 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Upstream-Bug
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue/Question

Under salt-cloud 2019.2, using script_args at provider level to pass options -j json_object to bootstrap-salt fails because openstacksdk tries to apply a python formatting to all config options and curly braces in json object fails python formatting.
The culprit line of openstacksdk code is here

Escaping curly-braces does not work.
Doubling them to escape python-formatting let you instanciate VM but this fails later on when executing bootstrap-salt because this is not a valid json object anymore.

Workaround is to move the script: and script_args configuration at profile level but, hey, who wants to duplicate configuration in each profile if they are the same everywhere?

Setup

Note: I don't use /etc/openstack/cloud.yml but rather define all provider onfiguration in /etc/salt/cloud.provider.d/<provider>.conf
Note2: If relevant, I use OVH public cloud, based on openstack

Using salt-cloud 2019.2 on Debian 9.

/etc/salt/cloud.providers.d/ovh-gra3.conf:

ovh-gra3:
  driver: openstack
  region_name: GRA3
  profile: ovh
  auth:
    username: REDACTED
    password: 'REDACTED'
    project_id: REDACTED
    project_name: 'REDATED'

  ssh_interface: private_ips
  ssh_key_name: my_ssh_key
  ssh_key_file: /path/to/keyfile

  minion:
    master: salt-master

  script: bootstrap-salt
  script_args: |
    -j '{"master": "salt-master"}' stable 2019.2
	
  security_groups:
    - no-input

  nics:
    - net-id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx	
    - net-id: yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy

/etc/salt/cloud.providers.d/ovh-gra3.conf:

ovh-gra3_s1-2_deb9base:
  provider: ovh-gra3
  size: s1-2
  image: 'Debian 9'
  ssh_username: 'debian'
Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

# cat my-cloud.map 
ovh-gra3_s1-2_deb9base:
  - test-saltcloud.company.tld

# salt-cloud -m my-cloud.map 
The following virtual machines are set to be created:
  test-saltcloud.company.tld

Proceed? [N/y] y
... proceeding
[ERROR   ] Failed to create VM test-saltcloud.company.tld. Configuration value u'"master"' needs to be set
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1288, in create
    output = self.clouds[func](vm_)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/openstack.py", line 682, in create
    conn = get_conn()
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/openstack.py", line 320, in get_conn
    conn = shade.openstackcloud.OpenStackCloud(cloud_config=None, **vm_)
  File "/usr/local/lib/python2.7/dist-packages/shade/openstackcloud.py", line 146, in __init__
    cloud_config = config.get_one_cloud(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/openstack/config/loader.py", line 1066, in get_one
    config[key] = value.format(**config)
KeyError: u'"master"'
test-saltcloud.company.tld:
    ----------
Versions Report
# salt-cloud --versions-report
Salt Version:
            Salt: 2019.2.0
 
Dependency Versions:
 Apache Libcloud: 1.5.0
            cffi: 1.12.3
        cherrypy: Not Installed
        dateutil: 2.5.3
       docker-py: Not Installed
           gitdb: Not Installed
       gitpython: Not Installed
           ioflo: Not Installed
          Jinja2: 2.9.4
         libgit2: 0.24.5
         libnacl: Not Installed
        M2Crypto: 0.24.0
            Mako: Not Installed
    msgpack-pure: Not Installed
  msgpack-python: 0.4.8
    mysql-python: Not Installed
       pycparser: 2.19
        pycrypto: 2.6.1
    pycryptodome: Not Installed
          pygit2: 0.24.2
          Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
    python-gnupg: Not Installed
          PyYAML: 3.12
           PyZMQ: 16.0.2
            RAET: Not Installed
           smmap: Not Installed
         timelib: Not Installed
         Tornado: 4.4.3
             ZMQ: 4.2.1
 
System Versions:
            dist: debian 9.9 
          locale: UTF-8
         machine: x86_64
         release: 4.9.0-8-amd64
          system: Linux
         version: debian 9.9 

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the provider and profile configuration shown, then trace salt/cloud/clouds/openstack.py, especially get_conn, into openstack/config/loader.py at the cited line. Determine where provider-level script_args are passed through formatting, and consider the existing profile-level workaround when defining the expected behavior; the issue is resolved when the shown configuration can create a VM and execute bootstrap-salt with the JSON argument intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.