User configurable "variables" for repeatedly used elements

Is your feature request related to a problem? Please describe.

Today, I frequently need to repeat specific pieces of info in my config which are all the same. When I update them, I have to check I got all settings. For example, if I'm deploying my template in a new region, I have at least 3 region settings to update, plus the region names in relevant arns. This can lead to bugs when I don't check it correctly. I would like a way to declare something once and reuse it multiple times.

Describe the solution you'd like

• I should be able to declare variables in the YAML

version: "2"
my-pipeline:
  variables:
    ACCOUNT_ID: "123456789012"
    REGION: "us-west-2"
    APP_NAME: "Foo"

• I should be able to access these variables with a simple semantic like ${{VARIABLE_NAME}}. For example:

- opensearch:
      hosts: [ "https://${{APP_NAME}}-${{ENV}}.${{REGION}}.es.amazonaws.com" ]

• I should be able to compose variables. For example:

variables:
    ACCOUNT_ID: "123456789012"
    ROLE_NAME: osis-role
    ROLE_ARN: "arn:aws:iam::${{ACCOUNT_ID}}:role/${{ROLE_NAME}}"

version: "2"
my-pipeline:
  variables:
    ACCOUNT_ID: "123456789012"
    REGION: "us-west-2"
    APP_NAME: "Foo"
    ENV: "dev-andercj"
    IS_SLS: true
    ROLE_NAME: osis-role
    ROLE_ARN: "arn:aws:iam::${{ACCOUNT_ID}}:role/${{ROLE_NAME}}"
  source:
    dynamodb:
      tables:
        - table_arn: "arn:aws:dynamodb:${{REGION}}:${{ACCOUNT_ID}}:table/${{APP_NAME}}-${{ENV}}"
          stream: 
            start_position: "LATEST"
          export:
            s3_bucket: "osis-${{APP_NAME}}-${{ENV}}"
            s3_prefix: "/ddbexports/${{APP_NAME}}/${{ENV}}" 
      aws:
        region: "${{REGION}}"
        sts_role_arn: "${{ROLE_ARN}}"
  sink:
    - opensearch:
      hosts: [ "https://${{APP_NAME}}-${{ENV}}.${{REGION}}.es.amazonaws.com" ]
      index: ${{APP_NAME}}
      action: ${getMetadata("opensearch_action")}
      document_id: ${getMetadata("primary_key")}
      aws:
        sts_role_arn: ${{ROLE_ARN}}
        region: ${{REGION}}
        serverless: ${{IS_SLS}}
      dlq:
        s3:
          bucket: "osis-${{APP_NAME}}-${{ENV}}"
          key_path_prefix: "dlq"
          region: ${{REGION}}
          sts_role_arn: ${{ROLE_ARN}}

Open questions/other thoughts:

• I should be able to declare variables outside the scope of a pipeline and it is then "global"
• I should be able to declare variables within a given pipeline's scope and another pipeline cannot access those variables
• I should be able to set variables with a processor (i.e. set_variable). These would only be set for that event's processing; there should be no side effects for other events.
  • Today, you can kind of hack this by adding entries, but then I have to remember to not include them in the actual index.
• I should be able to specify a separate config file which can I can override my variables with (not in the primary YAML), so that I can avoid having to modify YAML in CI systems.

Describe alternatives you've considered (Optional)

Option 1: CloudFormation-style parameters

Use CloudFormation-like syntax [0]. In this, you'd specify Parameters at the top and reference them with a Ref: child-property.

My personal opinion is that I don't find that this fits the current style of DataPrepper config and it solves for a slightly different problem set, but I do believe it would effectively solve the problem.

Option 2: getVariable() function

Similar to proposal, but would use a new function, getVariable(string variableName) to access a variables constants.

- opensearch:
      hosts: [ "https://${getVariable(\"APP_NAME\"}}-${getVariable(\"ENV\"}}.${getVariable(\"REGION\"}.es.amazonaws.com" ]

I think we may want to support this as well, but getVariable is verbose and would less readable than ${{VARIABLE_NAME}} syntax proposed.

Option 3: $ prefix

Use getMetadata() to access the variables, but use a $ prefix to signal that it is a variable - getMetadata("$VARIABLE_NAME")

- opensearch:
      hosts: [ "https://${getMetadata(\"$APP_NAME\"}}-${getMetadata(\"$ENV\"}}.${getMetdata(\"$REGION\"}.es.amazonaws.com" ]

If we support the "@" prefix [1] for autogenerated metadata, then I think we may want to support this as well, but getVariable is verbose and would less readable than ${{VARIABLE_NAME}} syntax proposed.

Options 2 & 3 could be useful in the future if there is support for ternary operators where you'd need to access the variable within the scope of a ${}

Additional context

• [0]: CloudFormation Parameters
• [1]: https://github.com/opensearch-project/data-prepper/issues/3630

The issue names no files, tests, or entry points. Start by agreeing on the variable scopes, interpolation syntax, processor behavior, and external override semantics; done would require an approved design and corresponding implementation and tests.