redpanda-data / redpanda-data/benthos
In workflow, enrichment listed in apply are not replayed if they are listed in skipped
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 571
- Forks
- 120
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 18
Description
Hey, I made this PR to fix the issue that I currently have, it's related with this Slack Thread.
Currently have an issue, all enrichments listed in apply meta are not replayed. We added this processor to replay all enrichment, even skipped ones because, some of them can be skipped because of a dependencies issue with the failed one. We added this processor before the workflow:
- label: replay_skipped
switch:
- check: this.meta.enrichments.exists("skipped") && this.meta.enrichments.skipped.length() > 0
processors:
- label: replay_skipped_enrichments
bloblang: |-
root = this
root.meta.enrichments.apply = this.meta.enrichments.failed.keys().merge(this.meta.enrichments.skipped)
Our workflow looks like that
---
processor_resources:
- label: enrichments
workflow:
meta_path: meta.enrichments
order:
...
But, in the document resulted, the enrichments listed in apply are not replayed, only the one previously in failed is replayed. They are not listed in succeeded, they are in skipped and the field added by enrichment are not present.
I'm not expert in Go but in my PR, if I well understand, in function skipFromMeta, the if skipped section should be before the if apply one because the if apply will remove the enrichment from the list skipList filled by the if skipped section.
I wrote test to validate if we have the expected result and it's works 🍾
I run tests for others functions TestWorkflows, TestWorkflowsDeps, TestWorkflowsParallel and TestWorkflowsWithOrderResources
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating skipFromMeta and read the workflow tests named TestWorkflows, TestWorkflowsDeps, TestWorkflowsParallel, and TestWorkflowsWithOrderResources. Reproduce the case where apply contains previously skipped enrichments, then verify that those enrichments are replayed and appear in the resulting document, alongside the existing failed-enrichment behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100