Programatic use of benthos

To support others (me!) to build their applications on top of benthos, you would need access to some of the parts considered internal to benthos. There are some hooks you can use, namely the Streams API, cli.Run, Environment and the StreamsBuilder. However, things become more difficult if you want to run/test a set of processors or examine the ConfigSpec for a specific component.

Exposing certain key parts of benthos might make it a lot easier for others to integrate.

The following items might good candidates for being exposed:

• internal.component.processor.execute.go: ExecuteAll

Additionally, StreamBuilder could be extended to offer the ability to add/set components in a more go-like way, using structs instead of yaml content. Currently we have to resort to map[string]any{} and yaml.Marshal that, which works, but isn't great.

Start by reading internal.component.processor.execute.go, especially ExecuteAll, and trace the existing Streams API, cli.Run, Environment, and StreamsBuilder interfaces. Compare the current yaml.Marshal and map[string]any{} approach with the requested programmatic access, then define the exposed surface and verify that processors and ConfigSpec can be used as described.