elastic / elastic/integrations

ssi: system test coverage for integrations

Open
#13,453 5 comments 0 reactions 0 assignees View on GitHub
Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

Most of the elastic/security-service-integrations data streams have system tests. However, the tests do not always completely cover the code-specified behaviours for the data stream. For API endpoint-hitting data streams based on HTTP JSON and CEL the three levels of functionality that should be tested are:

0. **works at all**: at least one event can be ingested from the data stream's data source mock without reference to the expected count
1. **works reliably**: the number of events expected are reliably ingested from the data stream's data source mock
2. **paginates**: same as level 1. but an initial request gets a first set of events, paginates to a second set of events and then terminates in a third set of events; this tests start-up, pagination progression, and safe pagination completion
3. **progresses cursor**: same as level 2. but the data stream runs periodic requests more than once, using the cursor value obtained from the previous run when it is not the first

It should be the case that all our integration data streams satisfy at least level 0. although this is not the case because of [deployer limitations in elastic-package](https://github.com/elastic/elastic-package/issues/1302) which do not allow more than one deployer, so where we have a cloud sourced input and an API-based sourced input for a data stream, we cannot test both.

Where we have level 0. satisfied, we should — and can easily — promote the testing to level 1. by adding an `assert.hit_count` directive to the system test configuration.

Due to the behaviour of some APIs it is not possible to ensure that pagination is correctly operating in the data stream; level 2. This happens when time is part of the pagination logic because `elastic-package` does not have the capacity to run fake time to allow the testing to be completely reproducible. In some cases this _can_ hacked around, but it is not reliable and may require some imagination.

I do not believe that we can currently test packages at level 3. This would require a way to programmatically define the mock behaviour in a way that is not currently possible with the mocking system that we generally use github.com/elastic/stream.

We do have the capacity to calculate test coverage in at least CEL inputs, although this is not enabled in any integration, and there is no mechanism to automate the inclusion of the test coverage data in testing by `elastic-package` at this stage.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.