saltstack / saltstack/salt

[BUG] boto_rds, and `boto3.assign_funcs`, requires boto.

Open
#64,495 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

boto bug
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Custom execution module using boto3, specifically boto3.assign_funcs, requires boto, despite not having any boto dependencies.

Setup

Create a custom module, boto3_rds for example, and add the following code. Install only boto3 onto the minion.

"""
Connection module for Amazon RDS

.. versionadded:: 2015.8.0

:configuration: This module accepts explicit rds credentials but can also
    utilize IAM roles assigned to the instance through Instance Profiles.
    Dynamic credentials are then automatically obtained from AWS API and no
    further configuration is necessary. More Information available at:

    .. code-block:: text

        http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

    If IAM roles are not used you need to specify them either in a pillar or
    in the minion's config file:

    .. code-block:: yaml

        rds.keyid: GKTADJGHEIQSXMKKRBJ08H
        rds.key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs

    A region may also be specified in the configuration:

    .. code-block:: yaml

        rds.region: us-east-1

    If a region is not specified, the default is us-east-1.

    It's also possible to specify key, keyid and region via a profile, either
    as a passed in dict, or as a string to pull from pillars or minion config:

    .. code-block:: yaml

        myprofile:
            keyid: GKTADJGHEIQSXMKKRBJ08H
            key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
            region: us-east-1

:depends: boto3
"""
# keep lint from choking on _get_conn and _cache_id
# pylint: disable=E0602
# pylint whinging perfectly valid code
# pylint: disable=W0106


import logging

import salt.utils.versions
from salt.exceptions import SaltInvocationError

log = logging.getLogger(__name__)


# pylint: disable=import-error
try:
    # pylint: disable=unused-import
    import boto3

    # pylint: enable=unused-import
    from botocore.exceptions import ClientError

    logging.getLogger("boto3").setLevel(logging.CRITICAL)
    HAS_BOTO = True
except ImportError:
    HAS_BOTO = False
# pylint: enable=import-error


def __virtual__():
    """
    Only load if boto libraries exist and if boto libraries are greater than
    a given version.
    """

    return salt.utils.versions.check_boto_reqs(
        boto3_ver="1.3.1",
        check_boto=False,
        check_boto3=True,
    )


def __init__(opts):
    if HAS_BOTO:
        __utils__["boto3.assign_funcs"](__name__, "rds")


def describe_db_clusters(
    name=None,
    filters=None,
    jmespath="DBInstances",
    region=None,
    key=None,
    keyid=None,
    profile=None,
):
    """
    Return a detailed listing of some, or all, DB Clusters visible in the
    current scope.  Arbitrary subelements or subsections of the returned dataset
    can be selected by passing in a valid JMSEPath filter as well.

    CLI Example:

    .. code-block:: bash

        salt myminion boto_rds.describe_db_clusters jmespath='DBClusters[*].DBClusterIdentifier'

    """
    conn = _get_conn(region=region, key=key, keyid=keyid, profile=profile)
    pag = conn.get_paginator("describe_db_clusters")
    args = {}
    args.update({"DBClusterIdentifier": name}) if name else None
    args.update({"Filters": filters}) if filters else None
    pit = pag.paginate(**args)
    pit = pit.search(jmespath) if jmespath else pit
    try:
        return [p for p in pit]
    except ClientError as e:
        code = getattr(e, "response", {}).get("Error", {}).get("Code")
        if code != "DBClusterNotFound":
            log.error(__utils__["boto3.get_error"](e))
    return []

Run the example command in the docstring to test.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
Use above code blocks, add as custom module in /srv/salt/_modules, then run the example command in the describe_db_clusters function, targeting a minion with boto3 installed. Ensure no boto package is installed.

Expected behavior
Salt is using Python3.x and python2.x packages/libraries should be deprecated and no longer needed, except in special circumstances.

Screenshots
N/A

Versions Report

Salt Version:
          Salt: 3006.1

Python Version:
        Python: 3.10.11 (main, May  5 2023, 02:31:54) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: 1.6.4
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.12.1
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
        relenv: 0.12.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: amzn 2
        locale: utf-8
       machine: x86_64
       release: 4.14.311-233.529.amzn2.x86_64
        system: Linux
       version: Amazon Linux 2

Salt master/minion are similar in OS/Platform, packages, and versions.

Additional context
This module code is basically a copy of the boto_rds module, with a new function added. So I assume the issue is present in any boto_* module as well.

If you don't have boto installed, you will get the following error.

'boto3_rds' __virtual__ returned False: __init__ failed

As soon as you install boto, the custom module no longer produces an error.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the custom module under /srv/salt/_modules and the init call to boto3.assign_funcs, then reproduce the describe_db_clusters command with boto3 installed and boto absent. Done means a boto3-based custom module loads and runs without requiring the separate boto package.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.