Support parquet / composite dataformat OTel indices in the OpenSearch sink

Is your feature request related to a problem? Please describe.

The OpenSearch sink has built-in index_types that set up the index, alias, and mapping for us — log-analytics-plain (logs), trace-analytics-plain-raw (spans), and otel-v2-apm-service-map (service map).

These assume a normal Lucene cluster. We're sending the same logs/spans/service-map to a cluster running in OPTIMIZED mode (the pluggable dataformat feature — parquet primary, lucene secondary). Those indices need a few extra settings that the built-in templates don't have:

"pluggable.dataformat.enabled": true,
"pluggable.dataformat": "composite",
"composite.primary_data_format": "parquet",
"composite.secondary_data_formats": ["lucene"]

and a couple of mapping changes (spans need events/links as nested with dynamic:false, and attributes as flat_object). So we can't use the built-in index_types as they are.

Describe the solution you'd like

We'd like Data Prepper to support parquet/composite indices. What's the best way to onboard it? A few options we can think of:

• New index_types for parquet (e.g. log-analytics-optimized, trace-analytics-optimized-raw, otel-v2-apm-service-map-optimized) — same naming/ISM as today, just with the composite template.
• An option on the existing index_types (e.g. data_format: parquet) so one type can write either Lucene or composite settings.
• Something else that fits the sink better.

Describe alternatives you've considered (Optional)

Additional context

In our test branch (the observability-stack demo, dual-writing to a local parquet node) we skip the built-in index_type and point each sink at our own template file, which carries the composite settings and mapping changes above:

- opensearch:
    hosts: ["http://host.docker.internal:9200"]
    insecure: true
    index: "otel-spans-plain"
    template_type: index-template
    template_file: "/usr/share/data-prepper/templates/mustang-otel-spans-template.json"

Full pipeline and the three templates:

• pipelines.mustang.yaml
• mustang-otel-logs-template.json
• mustang-otel-spans-template.json
• mustang-otel-service-map-template.json

Happy to share more detail.

Start by locating the OpenSearch sink's built-in index_type handling and template definitions, then compare them with the linked pipelines.mustang.yaml and the three mustang-* template JSON files. Determine how parquet/composite settings and the span mapping changes should be exposed for logs, spans, and service maps, and validate the chosen configuration against those templates.