Unexpected behavior of memory buffer (Bug?)

Hello,
I have nats_jetstream input to read some messages from a stream and write the output to a SQLite database. I noticed that some messages are missing in the SQLite database. But probably the input and output component does not matter. The issue seems to be the buffer component.

It’s also possible that I may have misunderstood the documentation. If that’s the case, please consider clarifying or improving the buffer documentation.

I use redpanda-connect_4.67.5_windows_amd64, but the problem reproduces with older versions too (and also on Linux).

I have a simplified configuration:

logger:
  level: "trace"
  format: json
  add_timestamp: true

input:
  nats_jetstream:
    urls:
      - nats://localhost:4222
    stream: something
    deliver: all
    ack_wait: 5m
    max_ack_pending: 512

buffer:
  memory:
    limit: 1000

pipeline:
  processors:
    - label: "log_received_message"
      log:
        level: DEBUG
        message: "Received message"
        fields_mapping: |
          root.nats_subject = meta("nats_subject")
          root.nats_sequence_stream = meta("nats_sequence_stream")

output:
  stdout: {}

I have 1485 messages in the stream, but the amount does not matter I guess. I do not add more messages to the stream. If the component is started I expect that all 1485 messages are processed.

I see 1485 trace log messages like this: "Consumed 1 messages from 'nats_jetstream'."
I see 761 trace log messages: "label":"log_received_message"
I see 761 trace log messages like this: "Attempting to write 1 messages to 'stdout'."
I see 761 trace log messages like this: "Successfully wrote 1 messages to 'stdout'."

It does not matter how long I wait, the missing messages are not processed anymore. But all messages are acknowledged on the input. I confirmed this on Nats (no outstanding acks). Nothing changes if I terminate the app.

Why are 724 (1485 - 761) messages not processed by the pipeline? Where are the messages? What are they waiting for?

Start by running the simplified YAML configuration with 1,485 existing stream messages and compare the input, buffer, processor, and output trace counts. Trace the memory buffer behavior to determine where the 724 acknowledged messages remain; done means explaining the discrepancy and either correcting the behavior or clarifying the buffer documentation.