redpanda-data / redpanda-data/connect
Support mapping the contents in sync_response store
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
Currently, sync_response adds messages to an internal store and passes them back to the input. In the case of http_server input, this results in a multipart response. This isn't desirable in my use case:
input:
http_server:
path: /what-the-blob
allowed_verbs:
- POST
sync_response:
status: '${! meta("http_status_code").or(500) }'
pipeline:
processors:
# usually some stuff here
- mapping: |
#!blobl
meta http_status_code = @http_status_code.or(500)
root = if errored() { {"error": error()} } else { {"result": this} }
- sync_response: {} # capture the desired response here
output:
fallback:
- gcp_pubsub:
project: foo-project
topic: foo-topic
# if we get to this point then there are 2 messages
# in the store and we'll get a multipart response
- sync_response: {}
processors:
- mapping: |
#!blobl
meta http_status_code = 500
root = {"error": @fallback_error}
In this example, if we fail to publish a message then two messages are added to the sync response buffer and the response seems to be a 200. I'm trying to think of ways to solve this. One idea I wanted to bounce: Should sync_response take a mapping that gives the current state of the store and let you return the next state? The default config that matches today's behaviour:
sync_response:
mapping: |
root = this.store.concat([this.message])
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the sync_response input and output entry points and trace how messages are added to and returned from the internal store. Review the proposed mapping interface and its default mapping, then define behavior for selecting the next store state. Done means the mapping can control the captured response while preserving today's behavior by default, including the HTTP response case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100