redpanda-data / redpanda-data/connect
http processor errors
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
I recently started using Benthos and have been blown away so far, but have hit a wall while building a slightly more complicated section of the pipeline.
I'm trying to enrich from an HTTP service while caching it to reduce load on the service. The service can 404 if not available (the branch should skip), but it can also respond without the correct field (branch should skip also). The URL for the http request is interpolated with a key from the message.
When the http request fails it issues an error. I'm unsure as to how exactly it is failing.
An @service benthos log is emitted from the HTTP request
{"@timestamp":"", "@service":"benthos","component":"enrich_service.processor.1.0", "level":"ERROR", "message":"HTTP request to 'http://service/${! meta(\"msg_id\") }' failed: HTTP request returned unexpected response code (404): 404 Not Found"}
{"@timestamp":"", "@service":"benthos","component":"enrich_service.processor.1.1.0", "level":"WARN", "message":"getting from service", }
- I thought this shouldn't appear because it is 'caught', although the catch routine is still run, because my own log is emitted.
- The Benthos log's message displays an un-interpolated string. I couldn't tell if the interpolation was failing and that is what was requested, or whether it just logged the raw string. If that's the case, it would be useful to see the exact URL that was called.
I have a suspicion that the second catch is also catching cache get misses - I might need to errored = deleted() before the http request.
I was also wondering if there's a better way to handle boolean values in meta fields, as well as unset meta fields.
One option instead of the bracketed fallback is to have the opening request_map set them to the default value.
- label: "enrich_service"
branch:
# only branch for messages that have a msg_id to enrich with
request_map: |
root = this.msg_id | deleted()
meta msg_id = this.msg_id
processors:
- cache:
resource: the_cache
operator: get
key: ${! meta("msg_id") }
# catch empty cache => GET from http service and set if property set
- catch:
- http:
url: 'http://service/${! meta("msg_id") }'
verb: GET
- catch:
- log:
level: WARN
message: "getting from service"
fields:
key: '${! meta("msg_id") }'
meta: '${! meta() }'
- bloblang: 'meta failed_service = "failed"'
- branch:
request_map: 'root = json("property") | deleted()'
processors:
- cache:
resource: the_cache
operator: set
key: '${! meta("msg_id") }'
value: '${! content().number() }'
result_map: |
root.enriched = if (json("property") | false.bool()) && (meta("failed_service") | "") !="failed" {
json("property").number()
} else {
""
}
meta should_retry = if (meta("failed_service") | "") !="failed" {
false.bool()
} else {
true.bool()
}
[ ... ]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named. Start by reproducing the supplied YAML branch with the HTTP processor, cache get/set, and nested catch blocks, then trace the emitted errors and interpolation values. Done should document or test whether caught HTTP and cache errors log raw or interpolated URLs and how unset or boolean metadata behaves.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- yaml
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100