saltstack / saltstack/salt

Kubernetes state incorrectly reports deployment as Succeeded if namespace is missing

Open
#46,064 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Confirmed severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue/Question

It's possible to trick state salt.states.kubernetes.deployment_present to believe a new deployment was successful when it in fact was reported as a failure and nothing was actually created due to the fact that the namespace was missing.

Setup

Running Kubernetes cluster 1.9.2
Kubernetes python client 3

test.sls

Make sure namespace "idontexist" does NOT exist

Create nginx deployment:
  kubernetes.deployment_present:
    - name: nginx
    - namespace: idontexist
      metadata:
        name: nginx
        labels:
          name: nginx
      spec:
        replicas: 1
        template:
          metadata:
            labels:
              name: nginx
          spec:
            containers:
            - name: nginx
              image: nginx
              ports:
              - containerPort: 80
                name: http
Steps to Reproduce Issue

Run state and watch debug output

root@bofh:~# salt-call state.apply test -l debug
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/basic.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/basic.conf
[DEBUG   ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/basic.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/basic.conf
[DEBUG   ] Please install 'virt-what' to improve results of the 'virtual' grain.
[DEBUG   ] Connecting to master. Attempt 1 of 1
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'bofh', 'tcp://x.x.x.x:4506')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (10420)
[DEBUG   ] Setting zmq_reconnect_ivl to '10420ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'bofh', 'tcp://x.x.x.x:4506', 'clear')
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'bofh', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'bofh', 'tcp://x.x.x.x:4506')
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded state.apply
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'bofh', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'bofh', 'tcp://x.x.x.x:4506')
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] In saltenv 'base', looking at rel_path 'test.sls' to resolve 'salt://test.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/test.sls' to resolve 'salt://test.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://test.sls'
[DEBUG   ] No dest file found
[INFO    ] Fetching file from saltenv 'base', ** done ** 'test.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/base/test.sls
[DEBUG   ] Jinja search path: ['/var/cache/salt/minion/files/base']
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/test.sls' using 'jinja' renderer: 0.00288105010986
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/test.sls:
Create nginx deployment:
  kubernetes.deployment_present:
    - name: nginx
    - namespace: idontexist
      metadata:
        name: nginx
        labels:
          name: nginx
      spec:
        replicas: 1
        template:
          metadata:
            labels:
              name: nginx
          spec:
            containers:
            - name: nginx
              image: nginx
              ports:
              - containerPort: 80
                name: http

[DEBUG   ] LazyLoaded config.get
[DEBUG   ] Results of YAML rendering:
OrderedDict([('Create nginx deployment', OrderedDict([('kubernetes.deployment_present', [OrderedDict([('name', 'nginx')]), OrderedDict([('namespace', 'idontexist'), ('metadata', OrderedDict([('name', 'nginx'), ('labels', OrderedDict([('name', 'nginx')]))])), ('spec', OrderedDict([('replicas', 1), ('template', OrderedDict([('metadata', OrderedDict([('labels', OrderedDict([('name', 'nginx')]))])), ('spec', OrderedDict([('containers', [OrderedDict([('name', 'nginx'), ('image', 'nginx'), ('ports', [OrderedDict([('containerPort', 80), ('name', 'http')])])])])]))]))]))])])]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/test.sls' using 'yaml' renderer: 0.00677394866943
[DEBUG   ] LazyLoaded kubernetes.ping
[DEBUG   ] LazyLoaded kubernetes.deployment_present
[INFO    ] Running state [nginx] at time 11:02:46.092549
[INFO    ] Executing state kubernetes.deployment_present for [nginx]
[DEBUG   ] response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"deployments.extensions \"nginx\" not found","reason":"NotFound","details":{"name":"nginx","group":"extensions","kind":"deployments"},"code":404}

[DEBUG   ] response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"idontexist\" not found","reason":"NotFound","details":{"name":"idontexist","kind":"namespaces"},"code":404}

[INFO    ] {'spec': OrderedDict([('replicas', 1), ('template', OrderedDict([('metadata', OrderedDict([('labels', OrderedDict([('name', 'nginx')]))])), ('spec', OrderedDict([('containers', [OrderedDict([('name', 'nginx'), ('image', 'nginx'), ('ports', [OrderedDict([('containerPort', 80), ('name', 'http')])])])])]))]))]), 'metadata': OrderedDict([('name', 'nginx'), ('labels', OrderedDict([('name', 'nginx')]))])}
[INFO    ] Completed state [nginx] at time 11:02:46.197541 duration_in_ms=104.992
[DEBUG   ] File /var/cache/salt/minion/accumulator/140691299620304 does not exist, no need to cleanup.
[DEBUG   ] LazyLoaded config.option
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'bofh', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'bofh', 'tcp://x.x.x.x:4506')
[DEBUG   ] LazyLoaded highstate.output
[DEBUG   ] LazyLoaded nested.output
local:
----------
          ID: Create nginx deployment
    Function: kubernetes.deployment_present
        Name: nginx
      Result: True
     Comment:
     Started: 11:02:46.092549
    Duration: 104.992 ms
     Changes:
              ----------
              metadata:
                  ----------
                  name:
                      nginx
                  labels:
                      ----------
                      name:
                          nginx
              spec:
                  ----------
                  replicas:
                      1
                  template:
                      ----------
                      metadata:
                          ----------
                          labels:
                              ----------
                              name:
                                  nginx
                      spec:
                          ----------
                          containers:
                              |_
                                ----------
                                name:
                                    nginx
                                image:
                                    nginx
                                ports:
                                    |_
                                      ----------
                                      containerPort:
                                          80
                                      name:
                                          http

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 104.992 ms
Versions Report
Master
Salt Version:
           Salt: 2017.7.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-112-generic
         system: Linux
        version: Ubuntu 16.04 xenial
Minion
Salt Version:
           Salt: 2017.7.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-1049-aws
         system: Linux
        version: Ubuntu 16.04 xenial

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the kubernetes.deployment_present state implementation and reproduce the test.sls example with a missing namespace on the reported Kubernetes setup. Trace how the namespace-not-found response is handled; done means the state reports failure and no longer marks the deployment as changed or succeeded.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.