redpanda-data / redpanda-data/benthos

Feature Request: Add Public API for Processor Pre/Post Hooks to Aid Debugging

Open
#316 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
571
Forks
121
Avg merge
2d 1h
Merged PRs (30d)
18

Description

Hi,

Description

To improve the developer experience when writing and testing complex pipelines, one of my most requested features is the ability to properly debug a pipeline.

This would ideally include the ability to:

  • Step through each processor's execution
  • Run to a specific pipeline step and halt (Breakpoints)
  • Inspect the input message before a processor executes
  • Inspect the output message after a processor executes (very helpful for branch processors!)

Currently, there doesn't appear to be a public API that allows for this level of introspection, which is essential for building a step-through debugger or a visual pipeline tracer.

Alternatives Considered

My first approach was to try and implement this externally:

  • Parse an existing pipeline configuration
  • Manually wrap every single processor in a custom "debug" processor (which would record I/O or halt execution)
  • Feed this new, enriched pipeline configuration into a StreamBuilder.

This approach failed for two main reasons:

  • Complexity: Programmatically parsing and reliably wrapping all processors, especially nested ones (like in branch or switch processors), is extremely difficult and fragile.
  • Context Loss: This wrapping "mangles" the original processor paths (e.g., processors.0 becomes processors.0.debug_wrapper.original or something similar). This makes it very difficult to map the execution step back to the original pipeline configuration for visualization.

Proposed Solution

I request a new public API to inject a wrapper or hook around each processor during its construction, without altering the user-defined configuration path.

This could be conceptually similar to the existing BuildTraced() function on the StreamBuilder, but allows for custom pre- and post hook functions.
The pre- and post functions would provide informations about the wrapped processor (name, type, path, config - basically a WalkedComponent) and access to the currect message batch.
Using the return value of the hooks, one could influence the execution and stop the pipeline if desired.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the StreamBuilder's existing BuildTraced() entry point and the WalkedComponent information referenced in the proposal. Determine how processor construction and nested processors are represented before defining the public pre- and post-hook API. Done means the issue's requested introspection and pipeline-stopping behavior are supported without changing user-defined processor paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
developer-experience, devtools
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.