elastic / elastic/elastic-package
[Upgrade tests] Add support to run upgrade tests in policy tests
- Dominant language
- Go
- Stars
- 72
- Forks
- 141
- Avg merge
- 19h 42m
- Merged PRs (30d)
- 55
Description
Part of https://github.com/elastic/elastic-package/issues/1831
Add support to run as part of the policy tests, and upgrade test for each test defined in the package.
An upgrade test in the scope of policy tests consists of:
- Install a previous version of the package
- Create a package policy fulfilling the variables with the values found in the new package (development files).
- Install the version of the package that it is being worked.
- Upgrade the package policy.
- Perform validations (compare policy test after upgrading the package with the expected one).
- Delete package policy.
- Uninstall package.
In order to perform the upgrade of a package and the corresponding policy, these are the new steps that must be added to the current process:
1. Install package in the new version
```
POST kbn:/api/fleet/epm/packages/nginx/2.2.1
{
"force": true
}
```
2. Get the required package policy ID
- Search this ID via API
```
GET kbn:/api/fleet/package_policies
```
- It could be stored in the tests directly the package policy ID created in the previous step
3. Upgrade the given package policy ID, so the changes take effect, via API:
```
POST kbn:/api/fleet/package_policies/upgrade
{
"packagePolicyIds": ["6d916dff-281c-4a78-85a1-908f902e2d8e"]
}
```
## Considerations
### Differences with respect current policy tests
Currently, in policy tests the package to be tested is installed once at the beginning of the execution and `elastic-package` runs all the tests defined. Once finished all tests, the package is uninstalled.
Adding upgrade tests in these policy tests causes that this process must change since **for each test**:
- the old version must be installed and configured in a package policy.
- the new version must be installed and configured in the same package policy.
- this package must be uninstalled.
As this process must be performed for each test, it implies that **all these tests must run sequentially**.
This will cause that executing all the tests of a package will be slower.
### Upgrade tests with the same package version
There could be scenarios where the old and new package used in these upgrade tests are the same.
In these cases, the new version must contain a suffix added on the fly when building/installing it (e.g. `+`). Example: https://github.com/elastic/elastic-package/issues/1831#issuecomment-3570404676
This allows to avoid issues when package is installed in Fleet (reference https://github.com/elastic/elastic-package/issues/1831#issuecomment-3570086958).
### Add mechanism to disable tests
As there could be flakiness in those tests, there should be a new setting in the test configuration files to disable a specific test.
This could be an example of the settings to disable (TBD):
```yaml
upgrade_tests:
enable: false # Disable completely test
disable_until: 9.3.0 # Or, Disable upgrade test until a specific Elastic stack version
vars: ~
data_stream:
vars:
paths:
- "{{SERVICE_LOGS_DIR}}/access.log*"
```
## Requirements
- These upgrade tests must be **disabled by default**.
- These upgrade tests will focus on upgrading packages, and they are performed with the same Elastic stack version.
- Add new settings to disable upgrade tests in configuration test files.
- If those upgrade tests are not executed, the policy tests should execute the same steps as they do now
- Install the package just once at the beginning, run all the tests, and finally uninstall the package.
- If executed, results must be added as part of the `xUnit` results.
- Add a new flag and environment variable to select which version to use as the old one `--upgrade-from`:
- Values: `latest`, `previous` or a specific version (e.g. 2.0.1)
- Allow to define an environment variable to set the same value (e.g. `ELASTIC_PACKAGE_UPGRADE_TEST_FROM_VERSION`)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the existing policy-test execution process and its package install/uninstall lifecycle. Then inspect the test configuration handling, xUnit result generation, and CLI flag/environment-variable entry points for --upgrade-from and ELASTIC_PACKAGE_UPGRADE_TEST_FROM_VERSION. Done means upgrade tests are opt-in, preserve current behavior when disabled, and report their results in xUnit output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100