[BUG] OTLP traces - 403 Forbidden on HEAD /_alias/otel-v1-apm-span?...

Describe the bug
I have given Data Prepper all possible permissions on AWS Opensearch Service, but I am still getting those errors when starting up otlp tracing pipeline.

OTLP logs built from https://github.com/nazarewk/kmssap-opensearch-data-prepper/tree/284ab4052d51ce1dbcaef3dd1f1dc26d065d2b56 work perfectly fine (this is pretty much a rebase of https://github.com/opensearch-project/data-prepper/pull/1372 )

To Reproduce
Steps to reproduce the behavior (didn't confirm, but should do it):

1. Create Opensearch Service cluster

2. Create IAM Role with statement

         {
           "Sid" : "DataPrepperAll",
           "Effect" : "Allow",
           "Action" : [
             "es:ESHttp*",
           ],
           "Resource" : [
             "${data.aws_opensearch_domain.this.arn}/*",
           ],
         },
   

3. Create OpenSearch Role permissions:

   {
         cluster_permissions = [
           "cluster_all",
           "indices_all",
         ]
         index_permissions = [
           {
             index_patterns = ["otel-v1*"]
             allowed_actions = [
               "indices_all",
             ]
           }
         ]
       }
   

4. Assign OpenSearch role to the IAM Role

5. Assign IAM Role to Data Prepper instance

6. Configure Trace pipeline

   note i am replacing host and region during container startup

   ### START TRACES
   otel-trace-pipeline:
     workers: 2
     delay: "100"
     buffer:
       bounded_blocking:
         buffer_size: 512
         batch_size: 32
     source:
       otel_trace_source:
         # common
         port: 21890
         health_check_service: true
         thread_count: 200
         max_connection_count: 500
         # TODO: ssl
         ssl: false
         authentication:
           unauthenticated:
     sink:
       - pipeline:
           name: otel-trace-raw-pipeline
       - pipeline:
           name: otel-trace-service-map-pipeline
   otel-trace-raw-pipeline:
     workers: 2
     delay: "3000"
     buffer:
       bounded_blocking:
         buffer_size: 512
         batch_size: 64
     source:
       pipeline:
         name: otel-trace-pipeline
     processor:
       - otel_trace_raw:
       - otel_trace_group:
           # see https://github.com/opensearch-project/data-prepper/blob/a72025e028c6925436d55bd92ff2392b367155b5/data-prepper-plugins/opensearch/README.md
           hosts:
             - https://OPENSEARCH_HOST
           aws_region: OPENSEARCH_AWS_REGION
           aws_sigv4: true
     sink:
       - opensearch:
           # see https://github.com/opensearch-project/data-prepper/blob/a72025e028c6925436d55bd92ff2392b367155b5/data-prepper-plugins/opensearch/README.md
           insecure: false
           hosts:
             - https://OPENSEARCH_HOST
           aws_region: OPENSEARCH_AWS_REGION
           aws_sigv4: true
           index_type: trace-analytics-raw
   otel-trace-service-map-pipeline:
     workers: 2
     delay: "100"
     buffer:
       bounded_blocking:
         buffer_size: 512
         batch_size: 32
     source:
       pipeline:
         name: otel-trace-pipeline
     processor:
       - service_map_stateful:
           # The window duration is the maximum length of time the data prepper stores the most recent trace data to evaluvate service-map relationships.
           # The default is 3 minutes, this means we can detect relationships between services from spans reported in last 3 minutes.
           # Set higher value if your applications have higher latency.
           window_duration: 360
     sink:
       - opensearch:
           # see https://github.com/opensearch-project/data-prepper/blob/a72025e028c6925436d55bd92ff2392b367155b5/data-prepper-plugins/opensearch/README.md
           insecure: false
           hosts:
             - https://OPENSEARCH_HOST
           aws_region: OPENSEARCH_AWS_REGION
           aws_sigv4: true
           index_type: trace-analytics-service-map
   ### END TRACES
   
   ### START LOGS
   otel-logs-pipeline:
     workers: 2
     delay: "100"
     buffer:
       bounded_blocking:
         buffer_size: 512
         batch_size: 32
     source:
       otel_logs_source:
         # see https://github.com/opensearch-project/data-prepper/blob/00d7ec690ab8034a013f12b9fc0e78aa75087f82/data-prepper-plugins/otel-logs-source/README.md#L23-L23
         # common
         port: 21892
         health_check_service: true
         thread_count: 200
         max_connection_count: 500
         # TODO: ssl
         ssl: false
         authentication:
           unauthenticated:
     processor:
       - otel_logs_raw_processor:
     sink:
       - opensearch:
           # see https://github.com/opensearch-project/data-prepper/blob/a72025e028c6925436d55bd92ff2392b367155b5/data-prepper-plugins/opensearch/README.md
           insecure: false
           hosts:
             - https://OPENSEARCH_HOST
           aws_region: OPENSEARCH_AWS_REGION
           aws_sigv4: true
           index: otel-v1-logs-%{YYYY.ww}
   #    - file:
   #        # TODO: rotate this file
   #        path: /var/log/fluent-bit-forwarder/logs.json
   ### END LOGS
   

Expected behavior
Data Prepper has a functioning traces pipeline

Screenshots
...

Environment (please complete the following information):

• OS: Pod in EKS on Bottlerocket
• Version: https://github.com/nazarewk/kmssap-opensearch-data-prepper/tree/284ab4052d51ce1dbcaef3dd1f1dc26d065d2b56

Additional context

everything relevant is available above, mostly in foldable elements

Start with the opensearch sink and otel_trace_group configuration in the trace pipelines, then compare the trace setup with the working logs setup in the linked branch. Reproduce the HEAD request against the AWS OpenSearch Service configuration and verify that the traces pipeline starts without the reported 403 error.