elastic / elastic/integrations
Investigate Flaky System Test Failures: "Could not find the expected hits" Across Multiple Integrations
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
We have observed recurring flaky test failures in daily builds (and in some Pull Requests too), specifically with the error message:
> "test case failed: could not find the expected hits in ."
These failures are tracked under the [flaky-test label](https://github.com/elastic/integrations/issues?q=test%20case%20failed%3A%20could%20not%20find%20the%20expected%20hits%20label%3Aflaky-test)
The root cause is unclear and may require deeper analysis of the build environment, service access, and test orchestration.
Please investigate the underlying cause of these flaky test failures (environment setup, Elastic Agent logs accessing services,...).
## Observations
- **Tests pass locally**: The issue does not reproduce when running tests on local environments.
- **Resource constraints**: Initial suspicion was resource bottlenecks (CPU/memory) in the build environment, but GCP metrics do not show clear evidence of this.
- **Affected packages**: Many failing tests are for packages that create Terraform resources in system tests (e.g., aws, sentinel_one_cloud_funnel, trellix_edr_cloud, sublime_security, cisco_umbrella). However, some affected packages do not use Terraform in system tests (e.g., netskope, couchbase, google_workspace, network_traffic).
Contributor guide
Assessment
This issue has not been assessed yet.