[BUG] Data Prepper hung with data stuck in buffer

Describe the bug
Data Prepper is no longer processing data. There appears to be data stuck in the buffer. Buffer metrics are reporting bufferUsage of 75%. RecordsInFlight are 250,000 and records in buffer are 687,000+. The last message in my logs is

2023-04-25T20:42:43.344 WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Document [******] has failure.

java.lang.RuntimeException: Number of retries reached the limit of max retries(configured value 10)

Data prepper appears to be hung. I no longer see the source polling or any data flowing through the pipeline via the metrics.

To Reproduce
I have 2 out of 5 pipelines with this issue. All are pointing to the same domain. Around the time of this issue starting there was a brief write block on my domain. It is unclear if this issue is reproducible as 3 were able to continue to process data after the write block was lifted..
Steps to reproduce the behavior:

Expected behavior
Data prepper does not hang with data in the buffer.

Environment (please complete the following information):

• Version [e.g. 2.2]

Additional context
Partial Pipeline Configuration

version: "2"
my-pipeline:
  source:
    s3:
      notification_type: "sqs"
      buffer_timeout: "60s"
      codec:
        newline:
          skip_lines: 1
      sqs:
        queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/****-queue"
      compression: "gzip"
      aws:
        region: "us-east-1"
        sts_role_arn: "arn:aws:iam::123456789012:role/osis-pipeline-role"
  buffer:
    bounded_blocking:
      batch_size: 125000
      buffer_size: 1000000
...

  sink:
  - opensearch:
      max_retries: 10
      hosts:
      - "https://*****"
      index: "vpc-flow-logs-%{yyyy.MM.dd}"
      bulk_size: 20
      aws:
        region: "us-east-1"
        sts_role_arn: "arn:aws:iam::123456789012:role/*****-role"
  workers: 2
  delay: 0

Start with the OpenSearchSink failure path and the bounded_blocking buffer configuration shown in the report, including max_retries, batch_size, buffer_size, and workers. Trace what happens after retries are exhausted during a domain write block and verify the source resumes processing without leaving records stuck in the buffer; the payload names no test or source file to run.