Plugins Manager in 5.X and beyond.

We have implemented the plugin manager in logstash in 1.5 more than 2 years ago.
With the java changes in the core and our future plan I think its a good time to discuss
the state of plugin in logstash and how we handle the dependencies.

I want to see this as an open discussion, package manager / dependency are complex beast but it
shouldn't stop us from talking.
### First lets summarize the state of the plugins handling.
- We are using Bundler/rubygems as the control of the dependencies tree.
- We are using the `gem` as the data format
- Rubygems is hosting our plugins
- Plugins have cross dependencies (plugin **A** depends on plugin **B**)
- Some plugin are hybrid and have dependencies on java jars.
### Known issues
- Offline installing in a secure environment is still cumbersome, it requires a logstash instance to generate a "pack" of plugin
- xpack suffer from the same issues as the offline installation.
- Validating if its really a logtash plugin is fragile and slow. (require a rest call per plugin)
- Plugins requiring jars wont necessary mean they will use the exact version they want. `require_jar` doesn't offer a guarantee.
- conflict between cross dependencies in plugin still happen. (see Resolver class)
### Possible changes to bundlers and gemspec
- Checking all plugins to mark the dependencies correctly (runtime vs development)
- package signing would be nice to have and could remove the need to make rest call to check if its actually a logstash plugin?
- Implement a new bundler resolver to relax contrains checks.
- Implement our plugins manager as a bundler plugin?
### Open questions
- As we move forward with more java are `gem` and `bundler` still the right tools to handle dependencies?
- It is possible to Shadow gem dependencies into a single self contains artifact? It is worth it to help with xpack/offline?
- Is writing a simple installer that takes self contains tarballs a possible solution?

I understand the question are really open ended.

This was creating following a discussion @talevy and I had.