Support "content.runners_base_paths" in "get_runner"

When working on https://github.com/StackStorm/st2/pull/3762 (Inquiries Garbage Collection), I ran into an issue running integration tests. The way GC works in my PR is to set the status of the Inquiry to timeout, then run the post-run function of the inquiry, so that the workflow callback is performed. This is the same model that is already in use for Inquiry responses, just with a different status.

(virtualenv) vagrant@st2dev:~/st2$ ./st2reactor/bin/st2garbagecollector --config-file ~/st2/conf/st2.tests.conf

(...truncated...)

2017-10-04 03:07:00,762 ERROR [-] Failed to import module inquirer.
Traceback (most recent call last):
  File "/home/vagrant/st2/st2common/st2common/runners/base.py", line 49, in get_runner
    module = register_runner(module_name)
  File "/home/vagrant/st2/st2common/st2common/util/loader.py", line 182, in register_runner
    RUNNER_MODULES_CACHE[module_name] = imp.load_source(module_name, module_path)
IOError: [Errno 2] No such file or directory
2017-10-04 03:07:00,765 ERROR [-] Failed to delete inquiries: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/home/vagrant/st2/st2reactor/st2reactor/garbage_collector/base.py", line 241, in _fail_inquiries
    purge_inquiries(logger=LOG)
  File "/home/vagrant/st2/st2common/st2common/garbage_collection/inquiries.py", line 81, in purge_inquiries
    runner = runner_container._get_runner(runnertype_db, action_db, liveaction_db)
  File "/home/vagrant/st2/st2actions/st2actions/container/base.py", line 332, in _get_runner
    runner = get_runner(runnertype_db.runner_module)
  File "/home/vagrant/st2/st2common/st2common/runners/base.py", line 52, in get_runner
    raise ActionRunnerCreateError(e)

The config file shown in the arguments in the snippet above is what's used for the existing itests for garbage collection. It specifies the /tmp directory for the system.base_path option. Naturally, this is not where the runners are actually located at in a test setup.

The config option content.runners_base_paths seems to offer an alternative location for runners, but from what I can tell, this is only used during bootstrapping (i.e. st2ctl reload), and is not used within the get_runner function. Since system.base_pathseems to be the only way to tell my GC code where the runners are, I had to create a new config with that the system.base_path option set to st2tests/st2tests/fixtures/packs to allow the Inquiry GC integraiton tests to find the runner code.

Obviously creating a new config for each corner case in test doesn't scale. So I suggest we use the content.runners_base_paths optionally within the get_runner function, if it is configured.

TODO

• Add functionality to use content.runners_base_paths
• Update garbage collection integration tests to use st2.tests.conf instead of its own st2.tests2.conf (remove the latter)

Start with st2common/st2common/runners/base.py and trace how get_runner resolves runner modules, then inspect the content.runners_base_paths and system.base_path configuration used by the garbage-collection integration tests. Done means get_runner can use the configured runner paths and the tests use st2.tests.conf, with st2.tests2.conf removed.