elastic / elastic/elastic-package

Add assistance to detect common backwards compatibilty issues

Open
#579 7 comments 4 reactions 0 assignees View on GitHub
Team:Ecosystem
Dominant language
Go
Stars
72
Forks
141
Avg merge
19h 42m
Merged PRs (30d)
55

Description

- Relates https://github.com/elastic/elastic-package/issues/1831

It could be nice to have some feature to assist detecting backwards compatibility issues in packages. This could be part of `elastic-package check`, but it would have the particularity of needing to download (or checkout) previous versions of the package, so maybe it should be optional.

There are some things that could be automatically checked between versions:
* Incompatible changes in field data types.
* Removed fields.
* Changes in sets of dimensions.
* Changes in policy templates.
* Removed variables.
* Variables not referenced in any configuration.
* Removed data streams.
* If any of these issues are detected, check that there is at least a breaking change entry in the changelog.

Proposed solution
----
We will add a new `elastic-package test compatibility` subcommand. In general this command will be used to compare the compatibility of the definitions in two versions of a package. This subcommand would run tests decided by flags or the environment. It would have different modes:
* `[--from-git ] [--from-git-ref ]` Default execution would check if the current working copy is a git repository, with a branch forked from an upstream branch. If it is, it would checkout the package from the base commit to a temporary location and would compare with the local package. If it cannot obtain another package from the local repository, because it is not a repository or it doesn't contain a previous version of the package, the test does nothing. This could be extended with flags to select a source repository and commit.
* `--from-registry [--kibana-version ] [--all]` would compare the local package with versions published in the indicated registry. By default it would compare with the latest version published, but could be extended to support comparing with the package of a given kibana version, or with all packages.
* `--from-registry --package --from-registry-version --to-registry-version ` would compare package versions from the registry. It does not require a local copy. This could be used to check compatibility between versions already published.

For any of the modes, the compatibility checks would be performed in the following ways:
* Static checks based on comparing the definitions for each package, taking into account fields imported from ECS, `ecs@mappings` and so on. Comparisons would be done by looping over all definitions in the old package, one by one, and checking for the presence of a compatible definition in the new one. We would also need to look for removed definitions. This would be the default.
* `--system` System checks based on installing both package versions, dumping the generated assets, and comparing them following similar rules as the static checks. It would also do a dummy upgrade testing by installing one version after the other. The stack used would be the same as in other tests, the one in the current profile, or indicated by environment variables.

The specific checks that will be performed on each execution would be the ones mentioned in the top of the description. We will start by implementing the tests for field mappings compatibility, and would follow by implementing the ones about variables.

Packages will have the option to skip these tests following the common settings we have for that in other tests. Additionally, breaking change notices in the changelogs can be used to circunvent detected failures.

We would start implementing the local git mode, with static checks, what could be already useful to detect many issues in fields changes. Other modes and tests would be implemented on following phases.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.