redpanda-data / redpanda-data/connect

Redpanda output with idempotent_writes makes unordered results

Open
#4,387 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Hi Team

I have unordered results using redpanda input/output. Source topic has been replied 3 times, so it contains the same blocks of messages. For some reason, the 3rd block got disordered as on screenshot below.

Image

My input and output configuration looks like below

output:
    label: kafka_output
    processors:
        - label: remove_build_in_meta_added
          mapping: |-
            let blacklist = ["kafka_topic", "kafka_partition", "kafka_offset",
            "kafka_lag", "kafka_timestamp_unix", "kafka_tombstone_message",
            "backoff_duration", "retry_count", "schema_id"]

            meta = metadata().filter(kv -> !$blacklist.contains(kv.key))
    redpanda:
        seed_brokers:
            - output-kafka-bootstrap.svc.cluster.local:9092
        topic: ${! meta("topic") }
        key: ${! meta("kafka_key") }
        timestamp_ms: ${! meta("kafka_timestamp_ms") }
        partitioner: murmur2_hash
        max_message_bytes: 5MB
        compression: zstd
        metadata:
            include_patterns:
                - .*
        sasl:
            - username: streams
              password: ${sasl_password}
              mechanism: SCRAM-SHA-512
        tls:
            enabled: true
            skip_cert_verify: true
        idempotent_write: true
        max_in_flight: 1

input:
    label: kafka_input
    redpanda:
        seed_brokers:
            - input-kafka-bootstrap.svc.cluster.local:9092
        topics:
            - source-topic
        consumer_group: source-topic
        auto_replay_nacks: true
        sasl:
            - username: streams
              password: ${sasl_password}
              mechanism: SCRAM-SHA-512
        tls:
            enabled: true
            skip_cert_verify: true
        fetch_max_partition_bytes: 5MB
        fetch_max_bytes: 50MB
        transaction_isolation_level: read_committed
        max_yield_batch_bytes: 5MB

I am using red panda connect version v4.61.0

I use idempotent writes and max in flight 1, I understand there might be duplicates as this is not the same as exactly once semantic, but to my understanding this should ensure topic is effectively ordered by looking at given key, meaning that in my example last record should have value 4529.49 and not 2203.57

Can you please tell me if this configuration should ensure data is effectively ordered within the scope of the same key?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the behavior with the shown input.redpanda and output.redpanda configurations, focusing on idempotent_write, max_in_flight, partitioner, and the repeated source blocks. Trace the output ordering for messages sharing a key and establish whether the observed order is guaranteed; done means the ordering semantics are confirmed or the configuration issue is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.