redpanda-data / redpanda-data/connect

gcp_pubsub output: not recovering from failures when ordering key is enabled

Open
#2,935 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs investigation
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Scenario

A gcp_pubsub connector (v4.37.0) configured something like this:

input:
  kafka: # btw, this is why we can't move to kafka_franz: https://github.com/redpanda-data/connect/issues/2745
    addresses: [ localhost:19092 ]
    topics: [ ordered-topic ]
    consumer_group: ordered-cg
    checkpoint_limit: 1 # https://docs.redpanda.com/redpanda-connect/components/inputs/kafka/#ordering
    batching:
      count: 100
      period: 1s
output:
  gcp_pubsub:
    project: <redacted>
    credentials: <redacted>
    topic: ordered-topic
    ordering_key: ordering-key

Expectation

After, for example, an internet interruption, the connector should automatically recover and resume publishing.

What we see

The connector does not auto-recover, and keeps failing with:

{"@service":"redpanda-connect","label":"","level":"error","msg":"Failed to send message to gcp_pubsub: pubsub: Publishing for ordering key, ordering-key, paused due to previous error. Call topic.ResumePublish(orderingKey) before resuming publishing","path":"root.output","stream":"stream-pubsub"}

you have to restart the process to restore the flow.

It looks like a call to topic.ResumePublish is missing, either after a failure or before each batch.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the gcp_pubsub output publishing path and inspect how failures are handled when an ordering key is configured. Compare that path with the Google Pub/Sub ResumePublish behavior described in the issue. Done means publishing automatically resumes after an interruption without restarting the process.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud
Domain
cloud, stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.