redpanda-data / redpanda-data/connect

Can you considering to replay all enrichments listed in skipped path for a workflow

Open
#4,422 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Hey, for the context, we currently have a workflow with many enrichment who call HTTP APIs. When an API doesn't answer, the document goes to a retry Kafka topic. When this document is replayed, the workflow will replay only the failed enrichment but not the others. In our case, some enrichments were ignored because they depended on the failed one. Consequently, several documents could have missing enrichments.

By reading the documentation, it's seems to be an expected behavior. We added an apply field to the meta_path who contain a list of all skipped and failed enrichments but it's doesn't works as expected. This field is just before the workflow but Benthos push the data inside the previous field and it's doesn't seems to care about the content.

So, is it possible to considering replay all enrichments listed in meta_path.skipped? (or maybe add an option to perform it)

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the workflow structured-metadata documentation and the retry Kafka topic replay path described in the issue. Trace how _meta_path.skipped and its apply field are handled when replayed, then define completion as replaying all skipped enrichments or documenting and implementing a clear option for that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.