[META] [FEATURE] Add a WorkflowStep for calling external REST APIs

Flow Framework allows sequencing of API calls, but these are presently limited to a fixed set of implemented steps. We can empower customers to perform automation and configuration using any OpenSearch or plugin API, or even calling external REST APIs.

Is your feature request related to a problem?

A workflow structure nearly identical to this plugin was proposed in this RFC for the Observability plugin: https://github.com/opensearch-project/observability/issues/1805

This API is designed to orchestrate multiple steps, each corresponding to a separate API call to a core API within OpenSearch. The primary goal of the Workflow API is to facilitate complex data preparation processes involving enrichment, aggregation, reindexing, and other transformations.

By enabling users to define multi-step workflows, this feature aims to simplify the preparation of raw data for visualization, thereby providing deeper insights into system status and health.

An example workflow step for an API call is included in the linked RFC:

"steps": [
{
  "name": "add_timestamp_field_to_otel_spans_index",
  "method": "PUT",
  "endpoint": "/${index_name}",
  "body": { ... }
 },
 ....
]

What solution would you like?

Add Workflow Steps which enable calling REST APIs.

The code already exists in our integration test classes, which could be refactored into appropriate steps in the main source tree:
https://github.com/opensearch-project/flow-framework/blob/13f672e1f210473802b292204bdd558963c9b871/src/test/java/org/opensearch/flowframework/TestHelpers.java#L81-L90

This is called with parameters like this:
https://github.com/opensearch-project/flow-framework/blob/13f672e1f210473802b292204bdd558963c9b871/src/test/java/org/opensearch/flowframework/FlowFrameworkRestTestCase.java#L135-L143

I think the best approach here is to write new steps:

• An InitHttpClientStep which creates or configures an appropriate client() to make the call. This would be set up very similar to how the ToolStep is created, using the workflow parameters to configure a Client object. https://github.com/opensearch-project/flow-framework/pull/530
• A RestApiStep which would reference the client in its previous node inputs, and take other params for the actual REST call.

One other consideration is the status API. We keep track of provisioned resources using step name, step id, resource type, and resource id. The linked RFC suggests intermediate status output with "complete" where we'd really want to identify what was done (hard to do generically so maybe we just put "complete") and for in-process steps, running tasks. We could consider that updated provisioning detail in a separate feature request.

We also need supporting code:

• Substitutions as indicated in comments below https://github.com/opensearch-project/flow-framework/pull/525
• The ability to wait for a particular state

What alternatives have you considered?

We could write workflow steps for every OpenSearch API (or at least the most common ones) and use the associated Transport Actions.

Do you have any additional context?

While making REST calls is less efficient than the transport calls, they enable calls outside of OpenSearch as well.

REST client calls may also be needed for future migration to serverless.

While relatively easy to do this with an HTTP Client, supporting HTTPS and preserving appropriate headers may introduce some security complexities.