Failure to render Configs before validation

SUMMARY

Configs do not render key/value objects before validations occur. Even though we are storing an array of urls we could change the config.schema to be an array but then we can not load from the key value store because when ST2 validates the config they do not load the key/value store values before doing the validation so it just sees that option as a string not as an array.

ISSUE TYPE

Bug Report

STACKSTORM VERSION

# st2 --version
st2 2.10.2, on Python 2.7.5

OS / ENVIRONMENT / INSTALL METHOD

Running on Rhel 7 using puppet module to install

STEPS TO REPRODUCE

config.schema:

---
encore_provision_sensor:
  description: Configuration for sensor to work
  type: object
  required: false
  additionalProperties: false
  properties:
    sn_server:
      description: "server urls"
      type: array
      required: true

config:

---
encore_provision_sensor:
  sn_server: "{{ st2kv.system.provision_sensor.servicenow.servers }}"

Keys store:

# st2 key get provision_sensor.servicenow.servers
+------------------+--------------------------------------------------------------+
| Property         | Value                                                        |
+------------------+--------------------------------------------------------------+
| name             | provision_sensor.servicenow.servers                          |
| value            | ["test.domain.tld", "test2.domain.tld",                      |
|                  | "test3.domain.tld"]                                          |
| secret           | False                                                        |
| encrypted        | False                                                        |
| scope            | st2kv.system                                                 |
| expire_timestamp |                                                              |
+------------------+--------------------------------------------------------------+

Error:

# st2ctl reload --register-configs
Registering content...[flags = --config-file /etc/st2/st2.conf --register-configs]
2019-03-01 15:55:15,693 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2019-03-01 15:55:15,703 INFO [-] Successfully connected to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2019-03-01 15:55:16,297 INFO [-] =========================================================
2019-03-01 15:55:16,297 INFO [-] ############## Registering configs ######################
2019-03-01 15:55:16,297 INFO [-] =========================================================
2019-03-01 15:55:17,830 WARNING [-] Failed to register configs: Failed to register config "/opt/stackstorm/configs/encore.yaml" for pack "encore": Failed validating attribute "encore_provision_sensor.sn_server" in config for pack "encore" (/opt/stackstorm/configs/encore.yaml): '{{ st2kv.system.provision_sensor.servicenow.servers }}' is not of type u'array'

Failed validating u'type' in schema['properties'][u'encore_provision_sensor'][u'properties'][u'sn_server']:
    {u'description': u'Servicenow server to use',
     u'required': True,
     u'type': u'array'}

On instance[u'encore_provision_sensor'][u'sn_server']:
    '{{ st2kv.system.provision_sensor.servicenow.servers }}'
Traceback (most recent call last):
  File "/usr/bin/st2-register-content", line 22, in <module>
    sys.exit(content_loader.main(sys.argv[1:]))
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/content/bootstrap.py", line 408, in main
    register_content()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/content/bootstrap.py", line 391, in register_content
    register_configs()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/content/bootstrap.py", line 346, in register_configs
    raise e
ValueError: Failed to register config "/opt/stackstorm/configs/encore.yaml" for pack "encore": Failed validating attribute "encore_provision_sensor.sn_server" in config for pack "encore" (/opt/stackstorm/configs/encore.yaml): '{{ st2kv.system.provision_sensor.servicenow.servers }}' is not of type u'array'

Failed validating u'type' in schema['properties'][u'encore_provision_sensor'][u'properties'][u'sn_server']:
    {u'description': u'Servicenow server to use',
     u'required': True,
     u'type': u'array'}

StackTrace:
https://github.com/StackStorm/st2/blob/master/st2common/st2common/content/bootstrap.py#L336
https://github.com/StackStorm/st2/blob/master/st2common/st2common/bootstrap/configsregistrar.py#L119
https://github.com/StackStorm/st2/blob/master/st2common/st2common/models/api/pack.py#L291
https://github.com/StackStorm/st2/blob/master/st2common/st2common/models/api/base.py#L71
https://github.com/StackStorm/st2/blob/master/st2common/st2common/util/schema/__init__.py#L305

Start with the registration path in st2common/content/bootstrap.py and st2common/bootstrap/configsregistrar.py, then follow validation through st2common/models/api/pack.py, st2common/models/api/base.py, and st2common/util/schema/init.py. Reproduce the st2ctl reload --register-configs case and verify that the st2kv reference is resolved before schema validation so the array-valued configuration passes.