[Meta] Make the Beats end-to-end tests pass consistently
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 364
Description
The end to end (E2E) tests for beats almost never pass (https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-mbp/job/main/), and are currently a quality gate for publishing beats snapshot artifacts (https://github.com/elastic/beats/pull/31135). A [large number of the E2E tests are flaky](https://github.com/elastic/e2e-testing/issues?q=is%3Aissue+is%3Aopen+label%3Aflaky-test), often for reasons unrelated to the Beats themselves. This is preventing beats snapshot builds from being published regularly.
The set of E2E tests to run for beats is defined [here](https://github.com/elastic/e2e-testing/blob/main/.ci/.e2e-tests-beats.yaml). The full set of available E2E tests can be found [here](https://github.com/elastic/e2e-testing/blob/main/.ci/.e2e-tests.yaml). The primary difference between the two sets of tests right now is that the system agent integration tests are disabled because of flakyness (https://github.com/elastic/e2e-testing/pull/2482).
The issue proposes the following high level plan to immediately address the issues with publishing beats snapshots and restore the E2E to a healthy state:
- [x] Revert https://github.com/elastic/beats/pull/31135 to keep the E2E tests running as part of the packaging stage, but do not require them to pass to publish snapshot artifacts.
- [x] https://github.com/elastic/beats/pull/32079
- [ ] Re-enable the system integration E2E tests that are currently disabled for beats but enabled for the full set of E2E tests that can run: https://github.com/elastic/e2e-testing/blob/24bbd614b2da10f6bed0da8b010e0f90c51b9eeb/.ci/.e2e-tests.yaml#L32-L34. At this point the tests will run but failures will not prevent publishing artifacts.
- [ ] Make the set of E2E tests currently configured to run for beats (defined [here](https://github.com/elastic/e2e-testing/blob/main/.ci/.e2e-tests-beats.yaml)) pass reliably, and then restore the behaviour where failing E2E tests prevent artifacts from being published.
Contributor guide
Assessment
This issue has not been assessed yet.