Local dataprepper DLQ file is not getting filled with dropped traces

Hi all,

I'm not sure if I'm missing something or if I'm doing something wrong on my end.

In my current data prepper setup there are traces that are getting dropped/sent to the logs of data prepper itself which pollutes its own logs and therefore important messages are getting swallowed in between all the dropped/logged traces. I would like to investigate those dropped traces further so that's why I added the local DLQ path to my pipeline config under the sink section so that those traces are getting written/exported into this file for further troubleshooting instead of the logs.

The problem that I'm facing now is that the DLQ file stays empty and is not being filled with the dropped traces. They are still getting dropped into the logs of data prepper. Data prepper is also not compaining about a misconfigured DLQ config in the logs. The DLQ file just stays empty for some reason.

The option to use AWS S3 for the DLQs is not working for me since we don't use it in our tech stack. We would either need support to use GCS or other 3rd party S3 options.

I found the config for the DLQ path here: https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/README.md

This is my current pipeline.yaml file:

otel-collector-pipeline:
   workers: 8
   delay: "100"
   source:
     otel_trace_source:
       ssl: false
       port: 9000
       health_check_service: true
   buffer:
     bounded_blocking:
       buffer_size: 25600
       batch_size: 400
   sink:
   - pipeline: 
       name: raw-pipeline
   - pipeline: 
       name: service-map-pipeline
raw-pipeline:
  workers: 8
  delay: "3000"
  source:
    pipeline:
      name: otel-collector-pipeline
  buffer:
    bounded_blocking:
      buffer_size: 25600
      batch_size: 3200
  processor:
  - otel_traces:
  - otel_trace_group:
      hosts: [ "https://api.of.our.opensearch.instance.com" ]
      username: dataprepper-user
      password: xxxxxxxxxxxxxxx
  sink:
  - opensearch:
      hosts: [ "https://api.of.our.opensearch.instance.com" ]
      username: dataprepper-user
      password: xxxxxxxxxxxxxxx
      index_type: trace-analytics-raw
      dlq_file: /usr/share/data-prepper/log/dlq-file
service-map-pipeline:
  workers: 8
  delay: "100"
  source:
    pipeline:
      name: otel-collector-pipeline
  processor:
  - service_map:
      window_duration: 180
  buffer:
    bounded_blocking:
      buffer_size: 25600
      batch_size: 400
  sink:
  - opensearch:
      hosts: [ "https://api.of.our.opensearch.instance.com" ]
      username: dataprepper-user
      password: xxxxxxxxxxxxxxx
      index_type: trace-analytics-service-map
      dlq_file: /usr/share/data-prepper/log/dlq-file

Maybe I misunderstand what the DLQ is/should be used for and I thought it would be suitable for my use case. I also don't know if it is expected behaviour for data prepper to write the complete traces into his own logs.

I'm using the newest release of data prepper (2.8.0)

I hope someone can shed some light on my problem and help me fix it.

If you need any additional information that I did not provide or forgot to provide, just ask and I will try my best to give you that information.

Cheers!

Start with data-prepper-plugins/opensearch/README.md and the supplied pipeline.yaml, using Data Prepper 2.8.0 configuration as the reproduction. Trace how dropped traces are handled by the OpenSearch sink and local dlq_file; done means the behavior is explained and the expected handling is documented or covered by a focused regression test.