elastic / elastic/package-spec
[Discuss] Support sequential package policy migrations across version upgrades
- Dominant language
- Go
- Stars
- 20
- Forks
- 93
- Avg merge
- 15h 10m
- Merged PRs (30d)
- 12
Description
## Problem
Today, when Fleet upgrades a package policy, it loads only the **target version's** package manifest and applies any `migrate_from` declarations found there. It does not walk through intermediate versions. This means if a user skips versions during an upgrade, migration definitions from intermediate versions are never applied and variable values can be silently lost.
**Example:**
- v1.0.0 has an `httpjson` input
- v2.0.0 replaces it with `cel` (declares `migrate_from: httpjson` on the `cel` input)
- v3.0.0 replaces `cel` with `otelcol` (declares `migrate_from: cel` on the `otelcol` input)
A user upgrading v1.0.0 → v3.0.0 will lose their configured variables, because v3.0.0 only knows to migrate from `cel`, not `httpjson`. The v2.0.0 migration that would have carried `httpjson` vars to `cel` is never seen.
This puts the burden on package authors to either:
1. Keep `migrate_from` declarations for all historical input types indefinitely (there's currently no guidance on when it's safe to remove them)
2. Hope that users never skip versions (not a safe assumption, especially for managed/auto-upgraded policies)
### How Kibana SO migrations handle this
For comparison, Kibana's saved objects migration system applies transforms **sequentially through every intermediate version**. Each version only describes its own delta, and the framework chains them automatically. A document at v1 being upgraded to v3 will have the v2 transform applied first, then v3. Migration authors never need to think about version skips.
(See `DocumentUpgradePipeline` in `src/core/packages/saved-objects/migration-server-internal/src/document_migrator/pipelines/upgrade_pipeline.ts` for the implementation.)
## Proposal
We could take inspiration from the SO migration system and have Fleet apply package policy migrations sequentially through intermediate versions.
The cumulative migration chain could be computed during the package build process (`elastic-package build`). The build tool already validates manifests against package-spec - it could also read `migrate_from` declarations from the current version and append them to the chain carried forward from the previous version. The result would be a `migrations` block baked into the built package archive, like:
```yaml
# v3.0.0 built package
migrations:
"2.0.0":
inputs:
cel:
migrate_from: httpjson
streams:
alert:
vars:
url:
migrate_from: request_url
"3.0.0":
inputs:
otelcol:
migrate_from: cel
```
Fleet would then read the policy's current package version, find all migrations between that version and the target, and apply them in order.
### Related issues
- https://github.com/elastic/kibana/issues/264085
- https://github.com/elastic/kibana/issues/266829
- https://github.com/elastic/kibana/issues/266831
- https://github.com/elastic/package-spec/issues/956
Contributor guide
Assessment
This issue has not been assessed yet.