How do I use the boto3 library correctly?

Hi there!

I'm flailing about trying to implement a salt-cloud plugin for AWS Lightsail, and I can't figure out how the heck to make all the import magic work.

Basically, I need access to the salt.utils.boto3.get_connection() call, in my plugin.

Searching the codebase for boto3, I can see it used extensively in various modules, but the ec2 cloud plugin seems to work entirely over HTTP(S) (I think?), which seems odd because I'm pretty sure you can control EC2 stuff using boto, and that seems much more "correct" then piecemeal HTTP stuff.

Anyways, if I import the boto3 module directly: import salt.utils.boto3, and then use it:

Traceback (most recent call last):
  File "/media/Storage/Scripts/salt/salt/cloud/clouds/lightsail.py", line 505, in _query
    conn = __get_connection()
  File "/media/Storage/Scripts/salt/salt/cloud/clouds/lightsail.py", line 494, in __get_connection
    conn = salt.utils.boto3.get_connection('lightsail')
  File "/media/Storage/Scripts/salt/salt/utils/boto3.py", line 207, in get_connection
    keyid, profile)
  File "/media/Storage/Scripts/salt/salt/utils/boto3.py", line 119, in _get_profile
    if not region and _option(service + '.region'):
  File "/media/Storage/Scripts/salt/salt/utils/boto3.py", line 100, in _option
    if value in __opts__:
NameError: global name '__opts__' is not defined

Apparently some of the mysterious import-magic is broken when it's used directly. Looking around for things that use it, it seems you have to set up the boto3... thing somehow with either a module level __init__() (how does that even work?), or including the setup call in your __virtual__() function.

The docs for the boto3.py say

import salt.utils.boto3

def __virtual__():
    # only required in 2015.2
    salt.utils.compat.pack_dunder(__name__)

    __utils__['boto.apply_funcs'](__name__, 'platform_name')

def test():
    conn = _get_conn()
    vpc_id = _cache_id('test-vpc')

Which is confusing, because I'm pretty sure the locations I've found using it refer to it as __utils__['boto3.assign_funcs'](__name__, 'platform_name'), so I guess the docs in boto3.py are just wrong? Anyways, putting either of the above (or the possible alternatives, boto3.assign_funcs, boto3.apply_funcs) in my __virtual__() call fails with a key error:

Traceback (most recent call last):
 File "/media/Storage/Scripts/salt/salt/loader.py", line 1676, in process_virtual
   virtual = getattr(mod, virtual_func)()
 File "/media/Storage/Scripts/salt/salt/cloud/clouds/lightsail.py", line 72, in __virtual__
   __utils__['boto3.apply_funcs'](__name__, 'lightsail')
 File "/media/Storage/Scripts/salt/salt/loader.py", line 1125, in __getitem__
   func = super(LazyLoader, self).__getitem__(item)
 File "/media/Storage/Scripts/salt/salt/utils/lazy.py", line 101, in __getitem__
   raise KeyError(key)
KeyError: 'boto3.apply_funcs'

Since elsewhere, the __utils__['boto3.assign_funcs'](__name__, 'platform_name') is located in a module level __init__() call, I tried that too, but you get the same key-error.

So, the above said, is it possible for my code to access the salt.utils.boto3 module, and if so, what's the correct way to do so? The enormous quantity of metaprogramming is breaking my tiny little brain.

For that matter, how does the stuff in __utils__ even work? It seems like it's mucking about in the available namespace on-the-fly, and functions and modules are getting dropped into the active local namespace somehow, which makes following anything almost impossible.

Start by tracing salt/cloud/clouds/lightsail.py into salt/utils/boto3.py, then inspect salt/modules/boto_asg.py and the loader.py handling of utils. Compare the documented boto3 initialization with the assign_funcs/apply_funcs references and determine the supported path; done means the plugin can access the connection without the reported NameError or KeyError, with the correct usage made clear.