redpanda-data / redpanda-data/connect

http processor errors

Open
#817 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

documentation ux
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

I recently started using Benthos and have been blown away so far, but have hit a wall while building a slightly more complicated section of the pipeline.

I'm trying to enrich from an HTTP service while caching it to reduce load on the service. The service can 404 if not available (the branch should skip), but it can also respond without the correct field (branch should skip also). The URL for the http request is interpolated with a key from the message.

When the http request fails it issues an error. I'm unsure as to how exactly it is failing.

An @service benthos log is emitted from the HTTP request

{"@timestamp":"", "@service":"benthos","component":"enrich_service.processor.1.0", "level":"ERROR", "message":"HTTP request to 'http://service/${! meta(\"msg_id\") }' failed: HTTP request returned unexpected response code (404): 404 Not Found"}
{"@timestamp":"", "@service":"benthos","component":"enrich_service.processor.1.1.0", "level":"WARN", "message":"getting from service", }
  1. I thought this shouldn't appear because it is 'caught', although the catch routine is still run, because my own log is emitted.
  2. The Benthos log's message displays an un-interpolated string. I couldn't tell if the interpolation was failing and that is what was requested, or whether it just logged the raw string. If that's the case, it would be useful to see the exact URL that was called.

I have a suspicion that the second catch is also catching cache get misses - I might need to errored = deleted() before the http request.

I was also wondering if there's a better way to handle boolean values in meta fields, as well as unset meta fields.
One option instead of the bracketed fallback is to have the opening request_map set them to the default value.

- label: "enrich_service"
  branch:
    # only branch for messages that have a msg_id to enrich with
    request_map: |
      root = this.msg_id | deleted()
      meta msg_id = this.msg_id
    processors:
      - cache:
         resource: the_cache
         operator: get
         key: ${! meta("msg_id") }
      # catch empty cache => GET from http service and set if property set
      - catch:
        - http:
            url: 'http://service/${! meta("msg_id") }'
            verb: GET
        - catch:
          - log:
              level: WARN
              message: "getting from service"
              fields:
                key: '${! meta("msg_id") }'
                meta: '${! meta() }'
          - bloblang: 'meta failed_service = "failed"'
        - branch:
            request_map: 'root = json("property") | deleted()'
            processors:
              - cache:
                  resource: the_cache
                  operator: set
                  key: '${! meta("msg_id") }'
                  value: '${! content().number() }'
    result_map: |
      root.enriched = if (json("property") | false.bool()) && (meta("failed_service") | "") !="failed" {
        json("property").number()
      } else {
        ""
      }
      meta should_retry = if (meta("failed_service") | "") !="failed" {
        false.bool()
      } else {
        true.bool()
      }
[ ... ]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named. Start by reproducing the supplied YAML branch with the HTTP processor, cache get/set, and nested catch blocks, then trace the emitted errors and interpolation values. Done should document or test whether caught HTTP and cache errors log raw or interpolated URLs and how unset or boolean metadata behaves.

Written by the indexing model from the issue text.

Assessment

Tech stack
yaml
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.