redpanda-data / redpanda-data/benthos

Feature Request: Add a render or preview command to show template output with specific config values

Open
#242 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

cli enhancement needs investigation
Dominant language
Go
Stars
571
Forks
121
Avg merge
2d 1h
Merged PRs (30d)
18

Description

Currently, there is no simple way to preview the fully rendered output of a Redpanda Connect template with a given set of config values. The existing workflows (rpk connect echo or rpk connect lint) either do not process templates or require writing test cases for every scenario.

Reference:
Slack thread: https://redpandacommunity.slack.com/archives/C075M18MAEA/p1749125364990089

Use Case:
As a developer, I want to quickly see what a template will render to with a specific config, for debugging and development purposes. This is especially useful when iterating on template logic or troubleshooting config issues, without having to write and maintain a test for every possible config variation.

command example:
rpk connect template render --template ./template.yaml --config ./config.yaml

template.yaml

name: kafka_input
type: input

fields:
  - name: broker_address
    type: string
    kind: list
  - name: topics
    type: string
    kind: list
  - name: cert_path
    type: string
  - name: consumer_group
    type: string

mapping: |
  root.input = { 
    "kafka": {
      "addresses" : this.broker_address,
      "topics": this.topics,
      "start_from_oldest": true,
      "consumer_group": this.consumer_group,
      "tls": {
        "enabled": true,
        "root_cas_file": "${this.cert_path}/ca.crt",
        "client_certs": [
          {
            "cert_file": "${this.cert_path}/user.crt",
            "key_file": "${this.cert_path}/user.key"
          }
        ]
      }
    }
  }

tests:
  - name: kafka_test
    config: 
      broker_address:
        - broker-1 
      topics:
        - topic-1
      cert_path: kafka-secrets
      consumer_group: app4134-composition-test
    expected:
      input:
        kafka:
          addresses:
            - broker-1 
          topics:
            - topic-1
          consumer_group: app4134-composition-test
          start_from_oldest: true
          tls:
            enabled: true
            root_cas_file: kafka-secrets/ca.crt
            client_certs:
              - cert_file: kafka-secrets/user.crt
                key_file: kafka-secrets/user.key

config.yaml

input:
  kafka_input:
    broker_address:
      - broker-1
    topics:
      - topic-1
    cert_path: demo-kafka-secret
    consumer_group: benthos-local-test-4

pipeline:
  processors:
    - mapping: |
        root.id = uuid_v4()
        root.foo = this.inner.foo
        root.body = this.outer`

expected output:

input:
  kafka:
    addresses:
      - broker-1
    topics:
      - topic-1
    start_from_oldest: true
    consumer_group: benthos-local-test-4
    tls:
      enabled: true
      root_cas_file: demo-kafka-secret/ca.crt
      client_certs:
        - cert_file: demo-kafka-secret/user.crt
          key_file: demo-kafka-secret/user.key

pipeline:
  processors:
    - mapping: |
        root.id = uuid_v4()
        root.foo = this.inner.foo
        root.body = this.outer

output:
  stdout: {}`

Benefit:
This would greatly improve the developer experience for anyone building or debugging templates, and would make Redpanda Connect templating more approachable and productive.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the existing rpk connect echo and rpk connect lint commands with the proposed rpk connect template render entry point. Trace how template.yaml and config.yaml should be loaded and how the fully rendered configuration should be emitted; done means the example produces the expected output without requiring a test case.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.