Let's Talk About Composability and Isolation (e.g. Inter-Pipeline Comms)

We now have a multiple pipelines feature.

Users already rig logstash->logstash communications up for a variety of purposes. Linking individual pipelines is something users will want to do as well.

In a broader sense the problems these users are solving by linking pipelines together can best be, in my estimation, encapsulated in two terms:

- **Composition:** Structured programming (if / else conditions) can be cumbersome without additional organizational units. In most programming languages methods or functions serve this purpose. When linking pipelines together the pipeline can perform this same task, somewhat like the actor model.
- **Isolation:** Pipelines have individually configurable performance criteria and are decoupled from each other by queues, which are also individually configurable. This is great when one has a forked flow and a filter plugin that interacts with an external system (like a JDBC lookup), since it can isolate performance issues with different queue / concurrency settings.

The key question of this issue is: **Are these problems best solved by inter-pipeline communications**?

Some alternate approaches:

1. Allow users to create `subpipelines` within pipelines.
1. Don't try to solve composition, just solve isolation by automatically scheduling / distributing work
1. Don't try to solve isolation, but add a notion of functions into logstash.
1. ``

An initial approach at solving this with our current infrastructure is present here: https://github.com/logstash-plugins/logstash-integration-internal/pull/1

Hypothetical configurations for it are presented here as thought exercises for its use: https://gist.github.com/andrewvc/b8c31706f8b6c8d5c5a3750643247832

I'd like to focus, however, on the big questions I've presented here before we move forward with that.

I think its likely that we may come to a conclusion of "we don't know". In that case I think it could be a useful exercise to release the `internal` input/output as non-bundled plugins and let users play with them, learning from real-world use cases what we should build.