saltstack / saltstack/salt

[BUG] salt-cloud naively assumes winrm commands succeed (and so deployments fail silently)

Open
#61,780 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Salt-Cloud Windows
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Using salt-cloud 3004 to deploy windows 2016 in EC2 and deployment always fails to deploy the minion but the job succeeds.

Aside: I haven't figured out why the minion fails to deploy yet but this issue is more about the lack of debuggability when it does

Setup

general setup instructions/script 
read -p "Subnet ID? " SUBNETID
read -p "Security Group ID"  SECURITY_GROUP_ID

# cat <<EOF > /etc/salt/cloud.providers.d/ec2.conf
ec2:
  id:  'use-instance-role-credentials'
  key:  'use-instance-role-credentials'
  keyname: se-demo-aws
  # securitygroup: default
  private_key: /etc/salt/pki/cloud/mykey.pem
  location: us-west-2
  ssh_interface: private_ips
  del_root_vol_on_destroy: True
  driver: ec2
  sync_after_install: all
EOF

cat <<EOF > /etc/salt/cloud.profiles.d/ec2.conf
ec2-w2016-us-west-2:
  provider: ec2
  size: t2.large
  userdata_file: /etc/salt/tools/windows-firewall.ps1
  location: us-west-2
  image: "Windows_Server-2016-English-Core-Base-2022.02.10"
  private_key: /etc/salt/pki/cloud/mykey.pem
  use_winrm: true
  winrm_verify_ssl: false
  win_username: Administrator
  win_password: auto
  win_installer: /srv/minion-cache/Salt-Minion-3004-Py3-AMD64-Setup.exe
  smb_port: 445
  network_interfaces:
    - DeviceIndex: 0
      PrivateIpAddresses:
        - Primary: True
      #auto assign public ip (not EIP)
      AssociatePublicIpAddress: True
      SubnetId: $SUBNET_ID
      SecurityGroupId:
        - $SECURITY_GROUP_ID
EOF


mkdir -p /etc/salt/tools

echo "Stolen from https://docs.saltproject.io/en/latest/topics/cloud/windows.html"

cat <<EOF > /etc/salt/tools/windows-firewall.ps1
<powershell>
New-NetFirewallRule -Name "SMB445" -DisplayName "SMB445" -Protocol TCP -LocalPort 445
New-NetFirewallRule -Name "WINRM5986" -DisplayName "WINRM5986" -Protocol TCP -LocalPort 5986

winrm quickconfig -q
winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="300"}'
winrm set winrm/config '@{MaxTimeoutms="1800000"}'
winrm set winrm/config/service/auth '@{Basic="true"}'

$SourceStoreScope = 'LocalMachine'
$SourceStorename = 'Remote Desktop'

$SourceStore = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Store -ArgumentList $SourceStorename, $SourceStoreScope
$SourceStore.Open([System.Security.Cryptography.X509Certificates.OpenFlags]::ReadOnly)

$cert = $SourceStore.Certificates | Where-Object -FilterScript {
    $_.subject -like '*'
}

$DestStoreScope = 'LocalMachine'
$DestStoreName = 'My'

$DestStore = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Store -ArgumentList $DestStoreName, $DestStoreScope
$DestStore.Open([System.Security.Cryptography.X509Certificates.OpenFlags]::ReadWrite)
$DestStore.Add($cert)

$SourceStore.Close()
$DestStore.Close()

winrm create winrm/config/listener?Address=*+Transport=HTTPS `@`{CertificateThumbprint=`"($cert.Thumbprint)`"`}

Restart-Service winrm
</powershell>
EOF

Steps to Reproduce the behavior

Using the above config (with pip installed smbprotocol and winrm)...

# salt-cloud -p ec2-w2016-us-west-2 dkf-sse860-01-w2016-01 -l debug 2>&1 | tee  /tmp/salt-cloud-w2016.log

This succeeds in deploying the instance but it fails to install the minion. This is where it goes wrong...

pdb debug of the build 
> /usr/lib/python3.6/site-packages/salt/utils/cloud.py(1334)deploy_windows()
-> if use_winrm:
(Pdb) l
1329                "C$",
1330                conn=smb_conn,
1331            )
1332            import pdb
1333            pdb.set_trace()
1334 ->         if use_winrm:
1335                winrm_cmd(
1336                    winrm_session,
1337                    "c:\\salttemp\\{}".format(installer),
1338                    ["/S", "/master={}".format(master), "/minion-name={}".format(name)],
1339                )
(Pdb) n
> /usr/lib/python3.6/site-packages/salt/utils/cloud.py(1335)deploy_windows()
-> winrm_cmd(
(Pdb) s
> /usr/lib/python3.6/site-packages/salt/utils/cloud.py(1336)deploy_windows()
-> winrm_session,
(Pdb) s
> /usr/lib/python3.6/site-packages/salt/utils/cloud.py(1337)deploy_windows()
-> "c:\\salttemp\\{}".format(installer),
(Pdb) s
> /usr/lib/python3.6/site-packages/salt/utils/cloud.py(1338)deploy_windows()
-> ["/S", "/master={}".format(master), "/minion-name={}".format(name)],
(Pdb) s
--Call--
> /usr/lib/python3.6/site-packages/salt/utils/cloud.py(2399)winrm_cmd()
-> def winrm_cmd(session, command, flags, **kwargs):
(Pdb) l
2394            log.exception("Failed to execute command '%s'", logging_command)
2395        # Signal an error
2396        return 1
2397
2398
2399 -> def winrm_cmd(session, command, flags, **kwargs):
2400        """
2401        Wrapper for commands to be run against Windows boxes using WinRM.
2402        """
2403        log.debug("Executing WinRM command: %s %s", command, flags)
2404        r = session.run_cmd(command, flags)
(Pdb) n
> /usr/lib/python3.6/site-packages/salt/utils/cloud.py(2403)winrm_cmd()
-> log.debug("Executing WinRM command: %s %s", command, flags)
(Pdb) n
[DEBUG   ] Executing WinRM command: c:\salttemp\Salt-Minion-3004-Py3-AMD64-Setup.exe ['/S', '/master=ip-10-0-195-179.us-west-2.compute.internal', '/minion-name=dkf-sse860-01-w2016-01']
> /usr/lib/python3.6/site-packages/salt/utils/cloud.py(2404)winrm_cmd()
-> r = session.run_cmd(command, flags)
(Pdb) s
--Call--
> /usr/local/lib/python3.6/site-packages/winrm/__init__.py(37)run_cmd()
-> def run_cmd(self, command, args=()):
(Pdb) l
 32             username, password = auth
 33             self.url = self._build_url(target, kwargs.get('transport', 'plaintext'))
 34             self.protocol = Protocol(self.url,
 35                                      username=username, password=password, **kwargs)
 36
 37  ->     def run_cmd(self, command, args=()):
 38             # TODO optimize perf. Do not call open/close shell every time
 39             shell_id = self.protocol.open_shell()
 40             command_id = self.protocol.run_command(shell_id, command, args)
 41             rs = Response(self.protocol.get_command_output(shell_id, command_id))
 42             self.protocol.cleanup_command(shell_id, command_id)
(Pdb) n
> /usr/local/lib/python3.6/site-packages/winrm/__init__.py(39)run_cmd()
-> shell_id = self.protocol.open_shell()
(Pdb) n
[DEBUG   ] Starting new HTTPS connection (1): 10.0.110.7:5986
[DEBUG   ] https://10.0.110.7:5986 "POST /wsman HTTP/1.1" 200 1628
> /usr/local/lib/python3.6/site-packages/winrm/__init__.py(40)run_cmd()
-> command_id = self.protocol.run_command(shell_id, command, args)
(Pdb) n
[DEBUG   ] https://10.0.110.7:5986 "POST /wsman HTTP/1.1" 200 847
> /usr/local/lib/python3.6/site-packages/winrm/__init__.py(41)run_cmd()
-> rs = Response(self.protocol.get_command_output(shell_id, command_id))
(Pdb) n
[DEBUG   ] https://10.0.110.7:5986 "POST /wsman HTTP/1.1" 500 1320
[DEBUG   ] https://10.0.110.7:5986 "POST /wsman HTTP/1.1" 500 1320
[DEBUG   ] https://10.0.110.7:5986 "POST /wsman HTTP/1.1" 200 1119
> /usr/local/lib/python3.6/site-packages/winrm/__init__.py(42)run_cmd()
-> self.protocol.cleanup_command(shell_id, command_id)

Weirdly enough, I've done this a few times and when I step over the rs = Response() I get the http error 500s. But if I step into that call, it retursn a bunch of http 200's (timing issue?) but the minion doesn't install correctly.

Anyway, there's no error checking going on here and no output is available for us to debug.

Expected behavior

It should fail and tell users why, without them having to resort to pdb to figure it out.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
# salt-cloud --versions
Salt Version:
            Salt: 3004

Dependency Versions:
 Apache Libcloud: 3.2.0
            cffi: 1.9.1
        cherrypy: 5.6.0
        dateutil: Not Installed
       docker-py: Not Installed
           gitdb: 0.6.4
       gitpython: 1.0.1
          Jinja2: 2.11.1
         libgit2: Not Installed
        M2Crypto: 0.35.2
            Mako: Not Installed
         msgpack: 0.6.2
    msgpack-pure: Not Installed
    mysql-python: Not Installed
       pycparser: 2.14
        pycrypto: Not Installed
    pycryptodome: 3.11.0
          pygit2: Not Installed
          Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
    python-gnupg: Not Installed
          PyYAML: 3.13
           PyZMQ: 17.0.0
           smmap: 0.9.0
         timelib: Not Installed
         Tornado: 4.5.3
             ZMQ: 4.1.4

Salt Extensions:
          SSEAPE: 8.6.0.3

System Versions:
            dist: centos 7 Core
          locale: UTF-8
         machine: x86_64
         release: 3.10.0-1160.53.1.el7.x86_64
          system: Linux
         version: CentOS Linux 7 Core

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/utils/cloud.py at deploy_windows() and winrm_cmd(), then reproduce with the supplied salt-cloud command and debug logging. Trace how session.run_cmd() handles the WinRM response and compare the command result with the deployment outcome. Done means a failed Windows minion installation reports the WinRM failure or command output instead of appearing successful.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.