redpanda-data / redpanda-data/connect
The `fallback` output should stop trying to send messages to unreachable outputs
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
The following config will keep trying to send messages to the kafka_franz output if it's unavailable, although there's an stdout fallback output configured:
input:
generate:
mapping: |
root.id = count("foo_counter1")
interval: 0s
count: 3
pipeline:
threads: 1
processors:
- bloblang: |
root = match {
this.id == 1 => {"msg": "foo"}
this.id == 2 => {"msg": "bar"}
_ => throw("UNKNOWN ID")
}
root.id = this.id
output:
switch:
retry_until_success: true
cases:
- check: errored()
output:
stdout:
codec: lines
processors:
- log:
message: "errored message: ${! json()}"
- output:
fallback:
- kafka_franz:
seed_brokers:
- localhost:6666
topic: test
- stdout:
codec: lines
processors:
- log:
message: "successful message: ${! json()}"
logger:
level: DEBUG
format: json
add_timestamp: true
static_fields:
"@service": benthos
shutdown_timeout: 3s # Default 20s
From the Discord chat, it looks like the solution would be to try and detect connection loss via an enhancement.
LE: Might be nice to also have an exponential_backoff field which tells fallthrough to wait a bit more time before trying again to send a message to an output which is experiencing connectivity issues.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the provided YAML configuration and observing how the fallback output handles an unavailable kafka_franz output. Trace the fallback behavior and connection-loss handling from the relevant output entry points; done means unreachable outputs stop being retried when a fallback is available, with any backoff behavior explicitly defined and tested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100