[meta] Improvements in current Logstash testing infrastructure

The motivation of this issue is to track and discuss the logstash efforts to improvements our current testing efforts in regards of unit, integration and acceptance testing.
- Test should have proper organization, so it's easy for developers to know they should fulfil all necessary gaps before thinking the feature is testing complete.
- Coverage can help knowing there are missing parts.
- Test should be able to run locally and remotely. Developers should not have to rely on the CI infra, but they should be able to either run vs a local VM (for example to debug issues in specific distros), or vs their local environment (properly sandboxed).
- Invoking each kind of test should be a one command call, including acceptance and integration test.
- A set of integrations test, using common configuration should be created. This should be part of the logstash-core project and don't necessary have to use the logstash command interface, just create internal pipelines. This must be our [white box testing](https://en.wikipedia.org/wiki/White-box_testing#Advantages) efforts, and must be mostly focused on testing pipeline interaction with plugins. Some of this is already available inside our testing code base, but should be more clearly DRY'ed.
- Another set of integration test should be created where the integration is tested from the user point of view, this will include external components like elasticsearch, kafka, etc. In correlation with the acceptance test layer this will benefit us checking how LS is behaving. This will necessarily need a one command that deals with this external components, providing users of this test with simplicity. There has been a PoC experiment by @talevy at https://github.com/talevy/elastic-stack-tests, we shuold also leverage existing cross platform efforts.
- We should have complete acceptance test that run regularly (each commit/nightly/...). Acceptance test are basically automatic validations for our deliverables from the user point of view, so they should check for common user actions and behaviour. Related #4868 #2852 .

The initial plan to get this up and running would be.
- [ ] Organize internal testing code base as clean as possible so developers know which testing parts they should always fill.
- [ ] Provide internal helpers to facilitate the creation of internal pipelines and later on the retrieval of test expectations out of them.
- [ ] Create a [first iteration](https://en.wikipedia.org/wiki/Minimum_viable_product) of an acceptance testing layer, this will include testing:
- [ ] Plugin manager behaviour, including CLI flags. #5020 (user stories)
- [ ] Standard apache config.
- [ ] `bin/logstash` CLI flags.
- [ ] Package installation and upgrade behaviours.

All this should be tested in all supported OS, and by using the generated packages from our codebase (aka snapshots).
- [ ] Create/investigate on ways of spinning up external services to interact with things like kafka, elasticsearch, redis, etc..
- [ ] Create a [first iteration](https://en.wikipedia.org/wiki/Minimum_viable_product) of an integration testing layer (black box), this could be focused on testing vs elasticsearch, or kafka output.
- [ ] Develop a way of running the integration test either locally or remotely.
- [ ] Develop a way of running the acceptance test either locally or remotely. (related to previous one)

_Some random final notes_:

Some this points don't need a fresh start as they are, in part, already there, but we need a more organized way of being sure we use them regularly.

If you are working on one of this task, please add your name on it.

Please attach issues/pulls/etc to each task so we're able to track the evolution of this here.

I tried to keep this only focused on behaviour and what we need/want for our testing infra, specific technology decisions should be keep in specific issues for each task.

And last but not least, this is not a complete issue, all your feedback is more than welcome!

This would not has been possible without the meaningful initial feedback from @talevy and @suyograo