[BUG] boto_rds, and `boto3.assign_funcs`, requires boto.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
Custom execution module using boto3, specifically boto3.assign_funcs, requires boto, despite not having any boto dependencies.
Setup
Create a custom module, boto3_rds for example, and add the following code. Install only boto3 onto the minion.
"""
Connection module for Amazon RDS
.. versionadded:: 2015.8.0
:configuration: This module accepts explicit rds credentials but can also
utilize IAM roles assigned to the instance through Instance Profiles.
Dynamic credentials are then automatically obtained from AWS API and no
further configuration is necessary. More Information available at:
.. code-block:: text
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
If IAM roles are not used you need to specify them either in a pillar or
in the minion's config file:
.. code-block:: yaml
rds.keyid: GKTADJGHEIQSXMKKRBJ08H
rds.key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
A region may also be specified in the configuration:
.. code-block:: yaml
rds.region: us-east-1
If a region is not specified, the default is us-east-1.
It's also possible to specify key, keyid and region via a profile, either
as a passed in dict, or as a string to pull from pillars or minion config:
.. code-block:: yaml
myprofile:
keyid: GKTADJGHEIQSXMKKRBJ08H
key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
region: us-east-1
:depends: boto3
"""
# keep lint from choking on _get_conn and _cache_id
# pylint: disable=E0602
# pylint whinging perfectly valid code
# pylint: disable=W0106
import logging
import salt.utils.versions
from salt.exceptions import SaltInvocationError
log = logging.getLogger(__name__)
# pylint: disable=import-error
try:
# pylint: disable=unused-import
import boto3
# pylint: enable=unused-import
from botocore.exceptions import ClientError
logging.getLogger("boto3").setLevel(logging.CRITICAL)
HAS_BOTO = True
except ImportError:
HAS_BOTO = False
# pylint: enable=import-error
def __virtual__():
"""
Only load if boto libraries exist and if boto libraries are greater than
a given version.
"""
return salt.utils.versions.check_boto_reqs(
boto3_ver="1.3.1",
check_boto=False,
check_boto3=True,
)
def __init__(opts):
if HAS_BOTO:
__utils__["boto3.assign_funcs"](__name__, "rds")
def describe_db_clusters(
name=None,
filters=None,
jmespath="DBInstances",
region=None,
key=None,
keyid=None,
profile=None,
):
"""
Return a detailed listing of some, or all, DB Clusters visible in the
current scope. Arbitrary subelements or subsections of the returned dataset
can be selected by passing in a valid JMSEPath filter as well.
CLI Example:
.. code-block:: bash
salt myminion boto_rds.describe_db_clusters jmespath='DBClusters[*].DBClusterIdentifier'
"""
conn = _get_conn(region=region, key=key, keyid=keyid, profile=profile)
pag = conn.get_paginator("describe_db_clusters")
args = {}
args.update({"DBClusterIdentifier": name}) if name else None
args.update({"Filters": filters}) if filters else None
pit = pag.paginate(**args)
pit = pit.search(jmespath) if jmespath else pit
try:
return [p for p in pit]
except ClientError as e:
code = getattr(e, "response", {}).get("Error", {}).get("Code")
if code != "DBClusterNotFound":
log.error(__utils__["boto3.get_error"](e))
return []
Run the example command in the docstring to test.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
Use above code blocks, add as custom module in /srv/salt/_modules, then run the example command in the describe_db_clusters function, targeting a minion with boto3 installed. Ensure no boto package is installed.
Expected behavior
Salt is using Python3.x and python2.x packages/libraries should be deprecated and no longer needed, except in special circumstances.
Screenshots
N/A
Versions Report
Salt Version:
Salt: 3006.1
Python Version:
Python: 3.10.11 (main, May 5 2023, 02:31:54) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: 1.6.4
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: 1.12.1
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 23.2.0
relenv: 0.12.3
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: amzn 2
locale: utf-8
machine: x86_64
release: 4.14.311-233.529.amzn2.x86_64
system: Linux
version: Amazon Linux 2
Salt master/minion are similar in OS/Platform, packages, and versions.
Additional context
This module code is basically a copy of the boto_rds module, with a new function added. So I assume the issue is present in any boto_* module as well.
If you don't have boto installed, you will get the following error.
'boto3_rds' __virtual__ returned False: __init__ failed
As soon as you install boto, the custom module no longer produces an error.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the custom module under /srv/salt/_modules and the init call to boto3.assign_funcs, then reproduce the describe_db_clusters command with boto3 installed and boto absent. Done means a boto3-based custom module loads and runs without requiring the separate boto package.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100