redpanda-data / redpanda-data/connect

Support mapping the contents in sync_response store

Open
#1,961 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

cool idea enhancement ux
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Currently, sync_response adds messages to an internal store and passes them back to the input. In the case of http_server input, this results in a multipart response. This isn't desirable in my use case:

input:
  http_server:
    path: /what-the-blob
    allowed_verbs:
      - POST
    sync_response:
      status: '${! meta("http_status_code").or(500) }'

pipeline:
  processors:
    # usually some stuff here
    
    - mapping: |
        #!blobl
        meta http_status_code = @http_status_code.or(500)
        root = if errored() { {"error": error()} } else { {"result": this} }

    - sync_response: {} # capture the desired response here

output:
  fallback:
    - gcp_pubsub:
        project: foo-project
        topic: foo-topic
    # if we get to this point then there are 2 messages
    # in the store and we'll get a multipart response
    - sync_response: {}
      processors:
        - mapping: |
            #!blobl
            meta http_status_code = 500
            root = {"error": @fallback_error}

In this example, if we fail to publish a message then two messages are added to the sync response buffer and the response seems to be a 200. I'm trying to think of ways to solve this. One idea I wanted to bounce: Should sync_response take a mapping that gives the current state of the store and let you return the next state? The default config that matches today's behaviour:

sync_response:
  mapping: |
     root = this.store.concat([this.message])

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the sync_response input and output entry points and trace how messages are added to and returned from the internal store. Review the proposed mapping interface and its default mapping, then define behavior for selecting the next store state. Done means the mapping can control the captured response while preserving today's behavior by default, including the HTTP response case.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.