Data Prepper - Logs - HTTP Source

Hi,

I was trying to create a log ingestion flow from OpenTelemetry to OpenSearch via DataPrepper.

So far, I've only seen otel_metrics_source and otel_trace_source in the DataPrepper documentation, is there a source that supports otlpexporter of OpenTelemetry Collector?

I went with using http source in DataPrepper in the end hoping it would work with otlphttp exporter, but so far I'm getting this error:

2023-01-13 14:14:10 2023-01-13T19:14:10.672Z    info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 1}
2023-01-13 14:14:10 2023-01-13T19:14:10.672Z    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"data-prepper:2021/v1/logs\": unsupported protocol scheme \"data-prepper\"", "interval": "3.465315768s"}
2023-01-13 14:14:25 2023-01-13T19:14:25.537Z    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"data-prepper:2021/v1/logs\": unsupported protocol scheme \"data-prepper\"", "interval": "14.167450749s"}
2023-01-13 14:14:39 2023-01-13T19:14:39.707Z    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"data-prepper:2021/v1/logs\": unsupported protocol scheme \"data-prepper\"", "interval": "29.737572211s"}

Below are my otel collector config and data prepper pipeline:

Otel Collector Config:

receivers:
  filelog:
    include: [ /etc/output/logs.log]

exporters:
  logging:
  otlphttp:
    endpoint: data-prepper:2021
    tls:
      insecure: true
      insecure_skip_verify: true

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [logging, otlphttp]

Data Prepper Pipeline:

logs-pipeline:
  source:
    http:
      ssl: false
      authentication:
        unauthenticated:
  processor:
    - grok:
        match:
          log: [ "%{COMMONAPACHELOG}" ]
  sink:
    - opensearch:
        hosts: [ "https://opensearch:9200" ]
        insecure: true
        username: admin
        password: admin
        index: apache_logs

I've tried changing log.log to log.json but I'm still getting the error.
I'm putting everything in docker compose and use docker to run all my containers.

Can someone please give me some information and point me to the right direction? Thank you very much for the help!

Start by reproducing the Docker Compose setup using the provided OpenTelemetry Collector configuration and Data Prepper logs-pipeline configuration. Inspect the HTTP source and OTLP HTTP request handling to determine whether the endpoint and payload are supported; completion should establish a working logs ingestion path or clearly document the missing support.