saltstack / saltstack/salt

[BUG] boto_ec2 - cant create/import keypairs - no workaround

Open
#59,856 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

boto bug severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

  1. You can't create an ec2 keypair, or import an existing one using any method
  2. The documentation for salt.states.boto_ec2.key_present is blank
  3. The documentation for salt.modules.boto_ec2.import_key is blank
  4. The documentation for salt.modules.boto_ec2.create_key is blank

There are nuances to importing keys, listed here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html, under "Option 2: Import your own public key to Amazon EC2"

When this does work, it would be of great benefit to tell users how to generate the keys in RFC4716 format. It was a new concept to me so it took me a bit to figure it out. A helpful example using ssh-keygen, in the docs, would have saved me significant time (ie - reduced my time to value).

Setup

#!/bin/bash
python3 -m pip install boto boto3 -q
mkdir -p /srv/salt/bug

cat <<EOF >/srv/salt/bug/show_bug.sls
/srv/salt/bug:
  file.directory:
    - create: True

Clean:
  cmd.run:
    - name: rm -f /srv/salt/bug/*.pem*
    - require:
        - /srv/salt/bug

Delete Key:
  boto_ec2.key_absent:
    - name: test0
    - region: eu-west-1
    - require:
        - Clean

# Let the module create the key
Create Key:
  boto_ec2.key_present:
    - name: test0
    - region: eu-west-1
    - save_private: /srv/salt/bug/
    - require:
        - Delete Key

/srv/salt/bug/generated.pem:
  cmd.run:
    - name: ssh-keygen -q -N '' -m pem -t rsa -b 4094 -f /srv/salt/bug/generated.pem
    - require:
        - Clean

/srv/salt/bug/generated.pem.pub.rfc4716:
  cmd.run:
    - name: "ssh-keygen -m RFC4716 -f /srv/salt/bug/generated.pem.pub -e > /srv/salt/bug/generated.pem.pub.rfc4716"
    - require:
        - /srv/salt/bug/generated.pem

/srv/salt/bug/generated.pem.pub.rfc4716.utf9:
  cmd.run:
    - name: "iconv -f ISO-8859-1 -t UTF-8 /srv/salt/bug/generated.pem.pub.rfc4716 > /srv/salt/bug/generated.pem.pub.rfc4716.utf9"
    - require:
        - /srv/salt/bug/generated.pem.pub.rfc4716

Import /srv/salt/bug/generated.pem.pub:
  boto_ec2.key_present:
    - name: test1
    - region: eu-west-1
    - upload_public: salt:///bug/generated.pem.pub
    - require:
        - /srv/salt/bug/generated.pem

Import /srv/salt/bug/generated.pem.pub.rc4716:
  boto_ec2.key_present:
    - name: test2
    - region: eu-west-1
    - upload_public: salt:///bug/generated.pem.pub.rfc4716
    - require:
        - /srv/salt/bug/generated.pem.pub.rfc4716

Import /srv/salt/bug/generated.pem.pub.rc4716.utf8:
  boto_ec2.key_present:
    - name: test3
    - region: eu-west-1
    - upload_public: salt:///bug/generated.pem.pub.rfc4716.utf8
    - require:
        - /srv/salt/bug/generated.pem.pub.rfc4716.utf9
EOF

Steps to Reproduce the behavior

*** Issue 1 ***

This call fails because base64.b64encode expects a bytes-like object instead of str; should be a simple fix

# salt-call boto_ec2.create_key module_test /srv/salt/bug/

Passed invalid arguments: a bytes-like object is required, not 'str'.
Usage:
    Creates a key and saves it to a given path.
    Returns the private key.

    CLI Example:
    .. code-block:: bash
        salt myminion boto_ec2.create_key mykey /root/

*** Issue 2 ***

This fails because save_path should be a directory.

  1. This shouldn't throw a stack trace, it should fail gracefully with an error
  2. The documentation of create_key should call this out (there is not documentation on the create_key params right now
# salt-call boto_ec2.create_key testkey /srv/salt/bug/filename
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
FileNotFoundError: [Errno 2] No such file or directory: '/srv/salt/bug/filename/testkey.pem'
Traceback (most recent call last):
  File "/bin/salt-call", line 11, in <module>
    load_entry_point('salt==3002.5', 'console_scripts', 'salt-call')()
  File "/usr/lib/python3.6/site-packages/salt/scripts.py", line 449, in salt_call
    client.run()
  File "/usr/lib/python3.6/site-packages/salt/cli/call.py", line 58, in run
    caller.run()
  File "/usr/lib/python3.6/site-packages/salt/cli/caller.py", line 112, in run
    ret = self.call()
  File "/usr/lib/python3.6/site-packages/salt/cli/caller.py", line 220, in call
    self.opts, data, func, args, kwargs
  File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 12, in execute
    return func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/modules/boto_ec2.py", line 1366, in create_key
    key.save(save_path)
  File "/usr/local/lib/python3.6/site-packages/boto/ec2/keypair.py", line 84, in save
    fp = open(file_path, 'wb')
FileNotFoundError: [Errno 2] No such file or directory: '/srv/salt/bug/filename/testkey.pem'
Traceback (most recent call last):
  File "/bin/salt-call", line 11, in <module>
    load_entry_point('salt==3002.5', 'console_scripts', 'salt-call')()
  File "/usr/lib/python3.6/site-packages/salt/scripts.py", line 449, in salt_call
    client.run()
  File "/usr/lib/python3.6/site-packages/salt/cli/call.py", line 58, in run
    caller.run()
  File "/usr/lib/python3.6/site-packages/salt/cli/caller.py", line 112, in run
    ret = self.call()
  File "/usr/lib/python3.6/site-packages/salt/cli/caller.py", line 220, in call
    self.opts, data, func, args, kwargs
  File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 12, in execute
    return func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/modules/boto_ec2.py", line 1366, in create_key
    key.save(save_path)
  File "/usr/local/lib/python3.6/site-packages/boto/ec2/keypair.py", line 84, in save
    fp = open(file_path, 'wb')
FileNotFoundError: [Errno 2] No such file or directory: '/srv/salt/bug/filename/testkey.pem'

*** Issue 3 ***

You can't import any existing keys because of the base64.b64encode issue. And if you do pass a utf8 encoded file to the "upload public" param, it doesn't recognize it as a valid file.

# salt-call state.apply bug.show_bug

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2106, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/states/boto_ec2.py", line 104, in key_present
    name, save_private, region, key, keyid, profile
  File "/usr/lib/python3.6/site-packages/salt/modules/boto_ec2.py", line 1366, in create_key
    key.save(save_path)
  File "/usr/local/lib/python3.6/site-packages/boto/ec2/keypair.py", line 85, in save
    fp.write(self.material)
TypeError: a bytes-like object is required, not 'str'

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2106, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/states/boto_ec2.py", line 115, in key_present
    name, upload_public, region, key, keyid, profile
  File "/usr/lib/python3.6/site-packages/salt/modules/boto_ec2.py", line 1394, in import_key
    key = conn.import_key_pair(key_name, public_key_material)
  File "/usr/local/lib/python3.6/site-packages/boto/ec2/connection.py", line 2935, in import_key_pair
    public_key_material = base64.b64encode(public_key_material)
  File "/usr/lib64/python3.6/base64.py", line 58, in b64encode
    encoded = binascii.b2a_base64(s, newline=False)
TypeError: a bytes-like object is required, not 'str'

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2106, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/states/boto_ec2.py", line 115, in key_present
    name, upload_public, region, key, keyid, profile
  File "/usr/lib/python3.6/site-packages/salt/modules/boto_ec2.py", line 1394, in import_key
    key = conn.import_key_pair(key_name, public_key_material)
  File "/usr/local/lib/python3.6/site-packages/boto/ec2/connection.py", line 2935, in import_key_pair
    public_key_material = base64.b64encode(public_key_material)
  File "/usr/lib64/python3.6/base64.py", line 58, in b64encode
    encoded = binascii.b2a_base64(s, newline=False)
TypeError: a bytes-like object is required, not 'str'

[ERROR   ] Unable to cache file 'salt://bug/generated.pem.pub.rfc4716.utf8' from saltenv 'base'.
[ERROR   ] You can either upload or download a private key
local:
----------
          ID: /srv/salt/bug
    Function: file.directory
      Result: True
     Comment: The directory /srv/salt/bug is in the correct state
     Started: 17:02:12.929803
    Duration: 7.522 ms
     Changes:
----------
          ID: Clean
    Function: cmd.run
        Name: rm -f /srv/salt/bug/*.pem*
      Result: True
     Comment: Command "rm -f /srv/salt/bug/*.pem*" run
     Started: 17:02:12.938416
    Duration: 9.937 ms
     Changes:
              ----------
              pid:
                  25733
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: Delete Key
    Function: boto_ec2.key_absent
        Name: test0
      Result: True
     Comment: The key test0 is deleted.
     Started: 17:02:13.057068
    Duration: 236.337 ms
     Changes:
              ----------
              old:
                  test0
----------
          ID: Create Key
    Function: boto_ec2.key_present
        Name: test0
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3.6/site-packages/salt/state.py", line 2154, in call
                  *cdata["args"], **cdata["kwargs"]
                File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2106, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/states/boto_ec2.py", line 104, in key_present
                  name, save_private, region, key, keyid, profile
                File "/usr/lib/python3.6/site-packages/salt/modules/boto_ec2.py", line 1366, in create_key
                  key.save(save_path)
                File "/usr/local/lib/python3.6/site-packages/boto/ec2/keypair.py", line 85, in save
                  fp.write(self.material)
              TypeError: a bytes-like object is required, not 'str'
     Started: 17:02:13.293840
    Duration: 247.981 ms
     Changes:
----------
          ID: /srv/salt/bug/generated.pem
    Function: cmd.run
        Name: ssh-keygen -q -N '' -m pem -t rsa -b 4094 -f /srv/salt/bug/generated.pem
      Result: True
     Comment: Command "ssh-keygen -q -N '' -m pem -t rsa -b 4094 -f /srv/salt/bug/generated.pem" run
     Started: 17:02:13.542144
    Duration: 1500.644 ms
     Changes:
              ----------
              pid:
                  25734
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: /srv/salt/bug/generated.pem.pub.rfc4716
    Function: cmd.run
        Name: ssh-keygen -m RFC4716 -f /srv/salt/bug/generated.pem.pub -e > /srv/salt/bug/generated.pem.pub.rfc4716
      Result: True
     Comment: Command "ssh-keygen -m RFC4716 -f /srv/salt/bug/generated.pem.pub -e > /srv/salt/bug/generated.pem.pub.rfc4716" run
     Started: 17:02:15.043600
    Duration: 13.798 ms
     Changes:
              ----------
              pid:
                  25737
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: /srv/salt/bug/generated.pem.pub.rfc4716.utf9
    Function: cmd.run
        Name: iconv -f ISO-8859-1 -t UTF-8 /srv/salt/bug/generated.pem.pub.rfc4716 > /srv/salt/bug/generated.pem.pub.rfc4716.utf9
      Result: True
     Comment: Command "iconv -f ISO-8859-1 -t UTF-8 /srv/salt/bug/generated.pem.pub.rfc4716 > /srv/salt/bug/generated.pem.pub.rfc4716.utf9" run
     Started: 17:02:15.058251
    Duration: 9.647 ms
     Changes:
              ----------
              pid:
                  25739
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: Import /srv/salt/bug/generated.pem.pub
    Function: boto_ec2.key_present
        Name: test1
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3.6/site-packages/salt/state.py", line 2154, in call
                  *cdata["args"], **cdata["kwargs"]
                File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2106, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/states/boto_ec2.py", line 115, in key_present
                  name, upload_public, region, key, keyid, profile
                File "/usr/lib/python3.6/site-packages/salt/modules/boto_ec2.py", line 1394, in import_key
                  key = conn.import_key_pair(key_name, public_key_material)
                File "/usr/local/lib/python3.6/site-packages/boto/ec2/connection.py", line 2935, in import_key_pair
                  public_key_material = base64.b64encode(public_key_material)
                File "/usr/lib64/python3.6/base64.py", line 58, in b64encode
                  encoded = binascii.b2a_base64(s, newline=False)
              TypeError: a bytes-like object is required, not 'str'
     Started: 17:02:15.068360
    Duration: 179.634 ms
     Changes:
----------
          ID: Import /srv/salt/bug/generated.pem.pub.rc4716
    Function: boto_ec2.key_present
        Name: test2
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3.6/site-packages/salt/state.py", line 2154, in call
                  *cdata["args"], **cdata["kwargs"]
                File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2106, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/states/boto_ec2.py", line 115, in key_present
                  name, upload_public, region, key, keyid, profile
                File "/usr/lib/python3.6/site-packages/salt/modules/boto_ec2.py", line 1394, in import_key
                  key = conn.import_key_pair(key_name, public_key_material)
                File "/usr/local/lib/python3.6/site-packages/boto/ec2/connection.py", line 2935, in import_key_pair
                  public_key_material = base64.b64encode(public_key_material)
                File "/usr/lib64/python3.6/base64.py", line 58, in b64encode
                  encoded = binascii.b2a_base64(s, newline=False)
              TypeError: a bytes-like object is required, not 'str'
     Started: 17:02:15.248298
    Duration: 174.846 ms
     Changes:
----------
          ID: Import /srv/salt/bug/generated.pem.pub.rc4716.utf8
    Function: boto_ec2.key_present
        Name: test3
      Result: False
     Comment: You can either upload or download a private key
     Started: 17:02:15.423780
    Duration: 121.756 ms
     Changes:

Summary for local
------------
Succeeded: 6 (changed=5)
Failed:    4
------------
Total states run:    10
Total run time:   2.502 s

Expected behavior
Be able to create a key using salt and have helpful documentation

Versions Report

Salt Version:
          Salt: 3002.5

Dependency Versions:
          cffi: 1.9.1
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 0.6.4
     gitpython: 1.0.1
        Jinja2: 2.11.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.0.0
         smmap: 0.9.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1160.15.2.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/modules/boto_ec2.py at create_key and import_key, then review salt/states/boto_ec2.py at key_present. Reproduce the reported Python 3 byte/string failures and the invalid save-path traceback using the supplied salt-call examples. Done means key creation and public-key import work, invalid paths fail gracefully, and the mentioned boto_ec2 documentation is no longer blank.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.