elastic / elastic/beats

[Meta] Make the Beats end-to-end tests pass consistently

Open
#32,076 1 comment 0 reactions 1 assignee Claimed by @leehinman View on GitHub
ci estimation:Week Team:Elastic-Agent-Data-Plane v8.5.0
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 2h
Merged PRs (30d)
364

Description

The end to end (E2E) tests for beats almost never pass (https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-mbp/job/main/), and are currently a quality gate for publishing beats snapshot artifacts (https://github.com/elastic/beats/pull/31135). A [large number of the E2E tests are flaky](https://github.com/elastic/e2e-testing/issues?q=is%3Aissue+is%3Aopen+label%3Aflaky-test), often for reasons unrelated to the Beats themselves. This is preventing beats snapshot builds from being published regularly.

The set of E2E tests to run for beats is defined [here](https://github.com/elastic/e2e-testing/blob/main/.ci/.e2e-tests-beats.yaml). The full set of available E2E tests can be found [here](https://github.com/elastic/e2e-testing/blob/main/.ci/.e2e-tests.yaml). The primary difference between the two sets of tests right now is that the system agent integration tests are disabled because of flakyness (https://github.com/elastic/e2e-testing/pull/2482).

The issue proposes the following high level plan to immediately address the issues with publishing beats snapshots and restore the E2E to a healthy state:

- [x] Revert https://github.com/elastic/beats/pull/31135 to keep the E2E tests running as part of the packaging stage, but do not require them to pass to publish snapshot artifacts.
- [x] https://github.com/elastic/beats/pull/32079
- [ ] Re-enable the system integration E2E tests that are currently disabled for beats but enabled for the full set of E2E tests that can run: https://github.com/elastic/e2e-testing/blob/24bbd614b2da10f6bed0da8b010e0f90c51b9eeb/.ci/.e2e-tests.yaml#L32-L34. At this point the tests will run but failures will not prevent publishing artifacts.
- [ ] Make the set of E2E tests currently configured to run for beats (defined [here](https://github.com/elastic/e2e-testing/blob/main/.ci/.e2e-tests-beats.yaml)) pass reliably, and then restore the behaviour where failing E2E tests prevent artifacts from being published.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.