Better support for logstash modules with logstash docker containers

@jordansissel I just wanted to create a ticket to follow up on our conversation that we started at the Berlin All Hands.

We have a dockerized version of the arcsight demo. In order to build that demo I spin up 3 build containers, one each for logstash, elasticsearch, and kibana. Once the logstash build container is up and running, I use docker exec to invoke a script inside the container that downloads the raw CEF data and pipes it through netcat to write to the logstash TCP pipeline.

What I expected to do was something like what I do elsewhere with beats to load the dashboards, where beats has a command that just does all the required setup and exits. However I ran into two issues:

1. You can't just run a parallel logstash command inside the docker image with the setup option, as it would try to use the same data directory and error out. I suppose we could point it at a separate data directory if necessary, but in this case the setup portion of writing dashboards doesn't even need a data directory so this seems like unnecessary hassle.

2. Even if you could do that, the logstash command runs indefinitely so there is no way for the build script to know when it was done loading the dashboards.

You can see how I worked around this issue for now by building a custom logstash image that has a separate start script to wait for elasticsearch and kibana containers to startup, and then runs the long-running logstash command line with the appropriate setup command line options as well.

https://github.com/elastic/demos/blob/master/arcsight/demo.sh#L43
https://github.com/elastic/demos/blob/master/arcsight/docker-compose-build.yml
https://github.com/elastic/demos/blob/master/arcsight/docker_ls/Dockerfile
https://github.com/elastic/demos/blob/master/arcsight/docker_ls/src/logstash.yml
https://github.com/elastic/demos/blob/master/arcsight/docker_ls/src/start_logstash.sh

It works, but just seems...harder than it should be. We could certainly just write up a blog post using this as an example for how people can do it, but I think it's probably possible to make things much more user friendly for this use case. I don't know what the right answer is and don't want to dictate the solution but I'll say that I don't think people should have to change the CMD and ENTRYPOINT in the Dockerfile to make it work, as that is not really standard practice and they would need to update that as command line arguments change over time instead of benefitting from our docker packaging keeping it up to date for them.

I'm happy to provide more information if necessary.

Start with the Docker-related workaround in the linked demos files: arcsight/demo.sh, docker-compose-build.yml, docker_ls/Dockerfile, logstash.yml, and start_logstash.sh. Read how the build containers wait for Elasticsearch and Kibana and invoke Logstash setup, then determine a supported container workflow that performs setup and exits without requiring custom CMD or ENTRYPOINT changes.

Written by the indexing model from the issue text.