redpanda-data / redpanda-data/benthos

In workflow, enrichment listed in apply are not replayed if they are listed in skipped

Open
#438 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
571
Forks
120
Avg merge
2d 1h
Merged PRs (30d)
18

Description

Hey, I made this PR to fix the issue that I currently have, it's related with this Slack Thread.

Currently have an issue, all enrichments listed in apply meta are not replayed. We added this processor to replay all enrichment, even skipped ones because, some of them can be skipped because of a dependencies issue with the failed one. We added this processor before the workflow:

- label: replay_skipped
  switch:
    - check: this.meta.enrichments.exists("skipped") && this.meta.enrichments.skipped.length() > 0
      processors:
        - label: replay_skipped_enrichments
          bloblang: |-
            root = this
            root.meta.enrichments.apply = this.meta.enrichments.failed.keys().merge(this.meta.enrichments.skipped)

Our workflow looks like that

---
processor_resources:
  - label: enrichments
    workflow:
      meta_path: meta.enrichments
      order:
...

But, in the document resulted, the enrichments listed in apply are not replayed, only the one previously in failed is replayed. They are not listed in succeeded, they are in skipped and the field added by enrichment are not present.

I'm not expert in Go but in my PR, if I well understand, in function skipFromMeta, the if skipped section should be before the if apply one because the if apply will remove the enrichment from the list skipList filled by the if skipped section.

I wrote test to validate if we have the expected result and it's works 🍾

I run tests for others functions TestWorkflows, TestWorkflowsDeps, TestWorkflowsParallel and TestWorkflowsWithOrderResources

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating skipFromMeta and read the workflow tests named TestWorkflows, TestWorkflowsDeps, TestWorkflowsParallel, and TestWorkflowsWithOrderResources. Reproduce the case where apply contains previously skipped enrichments, then verify that those enrichments are replayed and appear in the resulting document, alongside the existing failed-enrichment behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
stream-processing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.