redpanda-data / redpanda-data/connect
Can you considering to replay all enrichments listed in skipped path for a workflow
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
Hey, for the context, we currently have a workflow with many enrichment who call HTTP APIs. When an API doesn't answer, the document goes to a retry Kafka topic. When this document is replayed, the workflow will replay only the failed enrichment but not the others. In our case, some enrichments were ignored because they depended on the failed one. Consequently, several documents could have missing enrichments.
By reading the documentation, it's seems to be an expected behavior. We added an apply field to the meta_path who contain a list of all skipped and failed enrichments but it's doesn't works as expected. This field is just before the workflow but Benthos push the data inside the previous field and it's doesn't seems to care about the content.
So, is it possible to considering replay all enrichments listed in meta_path.skipped? (or maybe add an option to perform it)
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the workflow structured-metadata documentation and the retry Kafka topic replay path described in the issue. Trace how _meta_path.skipped and its apply field are handled when replayed, then define completion as replaying all skipped enrichments or documenting and implementing a clear option for that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100