Cannot specify st2kv keys in default values in pack config schemas

SUMMARY

Default values in pack configs that pull from the st2kv datastore are not interpolated with those values in sensors, and possibly actions.

STACKSTORM VERSION

Paste the output of st2 --version:

st2 3.2.0, on Python 3.6.8 running on CentOS 8

OS, environment, install method

Using st2vagrant with Virtualbox, installed normal open source StackStorm, and then used EWC upgrade script to install EWC.

Steps to reproduce the problem

In other words, specifying an st2kv will work as expected:

/opt/stackstorm/config/mypack.yaml

some_key: "{{ st2kv.system.mykey }}"

In sensor code:

    self.config['some_key']  # Returns the value from the st2kv datastore - 👍

but specifying that same value in the default value of a config schema does not work as expected:

/opt/stackstorm/packs/mypack/config.schema.yaml

some_key:
  type: string
  default: "{{ st2kv.system.mykey }}"

In sensor code:

    self.config['some_key']  # Returns the literal string '{{ st2kv.system.mykey }}' - 👎

Expected Results

The default value in the config schema should be interpolated with values from the st2kv datastore. So if the value in st2kv for the mykey key (and the system user) is myvalue, then myvalue should be the thing passed to the sensor (or action).

Actual Results

The config value, when read from the sensor's self.config dictionary, is the Jinja string literal, in this case {{ st2kv.system.mykey }}.

Start by tracing how packs/mypack/config.schema.yaml defaults are loaded and how the resulting value reaches a sensor's self.config; compare this with the working /opt/stackstorm/config/mypack.yaml path. Check the st2kv interpolation handling for config-schema defaults, including action configuration if applicable. Done means a default of "{{ st2kv.system.mykey }}" supplies the datastore value rather than the literal template.