elastic / elastic/integrations

[CI] Divide packages tested in serverless CI pipeline to be tested in batches

Open
#13,045 1 comment 0 reactions 0 assignees View on GitHub
ci Team:Ecosystem
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

Currently, the CI Pipeline in charge of testing the packages using a Elastic Serverless project (https://buildkite.com/elastic/integrations-serverless) creates one Serverless project (Observability or Security) and tests all those packages with that project/stack sequentally.

Given the current number of packages (~325), this pipeline started to last 4 hours. And this execution time will always be larger with any new package added/created.

That caused that the timeout required to be increased up to 6 hours (https://github.com/elastic/integrations/pull/12998).

This pipeline should be revisited in order to try to reduce the total time spent testing the packages with an Elastic Serverless project.

Possible approach (any other?):
- Split the number of packages in different batches (batches 100, 150 or 200 packages).
- Create dynamically a new Buildkite step for each batch defined and each step:
- Creates a new Elastic Serverless project.
- Tests all the packages in that batch sequentially.
- Deletes the Elastic Serverless project.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.