redpanda-data / redpanda-data/benthos
Feature Request: Add Public API for Processor Pre/Post Hooks to Aid Debugging
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 571
- Forks
- 121
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 18
Description
Hi,
Description
To improve the developer experience when writing and testing complex pipelines, one of my most requested features is the ability to properly debug a pipeline.
This would ideally include the ability to:
- Step through each processor's execution
- Run to a specific pipeline step and halt (Breakpoints)
- Inspect the input message before a processor executes
- Inspect the output message after a processor executes (very helpful for branch processors!)
Currently, there doesn't appear to be a public API that allows for this level of introspection, which is essential for building a step-through debugger or a visual pipeline tracer.
Alternatives Considered
My first approach was to try and implement this externally:
- Parse an existing pipeline configuration
- Manually wrap every single processor in a custom "debug" processor (which would record I/O or halt execution)
- Feed this new, enriched pipeline configuration into a StreamBuilder.
This approach failed for two main reasons:
- Complexity: Programmatically parsing and reliably wrapping all processors, especially nested ones (like in branch or switch processors), is extremely difficult and fragile.
- Context Loss: This wrapping "mangles" the original processor paths (e.g.,
processors.0becomesprocessors.0.debug_wrapper.originalor something similar). This makes it very difficult to map the execution step back to the original pipeline configuration for visualization.
Proposed Solution
I request a new public API to inject a wrapper or hook around each processor during its construction, without altering the user-defined configuration path.
This could be conceptually similar to the existing BuildTraced() function on the StreamBuilder, but allows for custom pre- and post hook functions.
The pre- and post functions would provide informations about the wrapped processor (name, type, path, config - basically a WalkedComponent) and access to the currect message batch.
Using the return value of the hooks, one could influence the execution and stop the pipeline if desired.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the StreamBuilder's existing BuildTraced() entry point and the WalkedComponent information referenced in the proposal. Determine how processor construction and nested processors are represented before defining the public pre- and post-hook API. Done means the issue's requested introspection and pipeline-stopping behavior are supported without changing user-defined processor paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- developer-experience, devtools
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100