elastic / elastic/package-spec

[Change Proposal] Move categories source of truth from package-registry to package-spec

Open
#1,157 0 comments 0 reactions 0 assignees View on GitHub
discuss enhancement
Dominant language
Go
Stars
20
Forks
93
Avg merge
15h 10m
Merged PRs (30d)
12

Description

Please read [Change Proposals in CONTRIBUTING.md](https://github.com/elastic/package-spec/blob/main/CONTRIBUTING.md#change-proposals). Thank you!

## Problem

The canonical category hierarchy (names, titles, parent/sub relationships) lives in `elastic/package-registry` at [`categories/categories.yml`](https://github.com/elastic/package-registry/blob/main/categories/categories.yml).

Meanwhile, `package-spec` duplicates the allowed category **keys** as a flat JSON-schema `enum` under `#/definitions/categories` in [`spec/integration/manifest.spec.yml`](https://github.com/elastic/package-spec/blob/main/spec/integration/manifest.spec.yml) (referenced by integration, input, and content manifests and by `policy_templates[].categories`). That duplicate list can drift from the YAML.

Separately, [#1095](https://github.com/elastic/package-spec/pull/1095) adds semantic validation that needs parent/sub relationships and currently performs an HTTP fetch against the registry YAML at validation time. That couples `elastic-package check` to the network and to another repo’s default branch.

## Proposal

1. Add `spec/categories.yml` to `package-spec`, ported verbatim from [`elastic/package-registry/categories/categories.yml`](https://github.com/elastic/package-registry/blob/main/categories/categories.yml).
2. Add a **public** Go package `code/go/pkg/categories` that `//go:embed`s `spec/categories.yml` and exposes an API equivalent to `package-registry/categories/` so the registry can adopt it in a follow-up (registry still needs to read this data from the spec package).
3. Remove the `enum:` block from `#/definitions/categories` in [`spec/integration/manifest.spec.yml`](spec/integration/manifest.spec.yml), leaving `categories` as string arrays at the schema level.
4. Add a semantic validator that loads `pkg/categories` and rejects any manifest `categories` value (top-level and under `policy_templates[]`) that is not a known category key in `spec/categories.yml`.
5. Include the [#1095](https://github.com/elastic/package-spec/pull/1095) follow-up in the same effort: delete `fetchRegistryParentCategories`, `packageRegistryCategoriesURL`, and any network-dependent tests; source parent/sub lookups from `pkg/categories` instead.

## Where this needs to be implemented

| Repo | Change | Order |
| --- | --- | --- |
| `elastic/package-spec` | Add `spec/categories.yml`, `code/go/pkg/categories`, remove schema `enum` at `#/definitions/categories`, add semantic validator for `categories`, retrofit [#1095](https://github.com/elastic/package-spec/pull/1095) to drop HTTP. Update [`spec/changelog.yml`](spec/changelog.yml), [`AGENTS.md`](AGENTS.md) if useful. | 1st |
| `elastic/package-registry` | Switch `categories/` to import/re-export `github.com/elastic/package-spec/v{major}/code/go/pkg/categories`; deprecate/remove duplicate YAML when ready. | 2nd |
| `elastic/elastic-package` | Bump `package-spec` dependency after release; no direct code changes expected. | 3rd |
| `elastic/kibana` | No runtime change: Fleet still loads facets via registry [`fetchCategories`](https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/server/services/epm/registry/index.ts) (`GET ${registryUrl}/categories`). Optional later: refresh [`PackageSpecCategory`](https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/common/types/models/package_spec.ts) when registry/spec agree on keys. | Later |

After consensus, this issue should gain an **ordered** checklist with links to filed issues in each repo per CONTRIBUTING.md step 3–4.

## Proposed implementation (pre-consensus)

The following is a concrete sketch for discussion before consensus and before splitting into tracked issues:

1. Add `spec/categories.yml` (port verbatim from [`elastic/package-registry/categories/categories.yml`](https://github.com/elastic/package-registry/blob/main/categories/categories.yml)).
2. Add `code/go/pkg/categories` with `//go:embed` of `spec/categories.yml`, parsing compatible with registry’s structure (parents + flattened subcategory keys), exported `Default()` / reader helpers suitable for registry reuse.
3. In [`spec/integration/manifest.spec.yml`](spec/integration/manifest.spec.yml), edit `#/definitions/categories`: keep `type: array` / `items.type: string`, remove the entire `enum:` list (single edit affects integration, input, content manifests and `policy_templates[].categories` via `$ref`).
4. Register a new semantic validator under [`code/go/internal/validator/semantic/`](code/go/internal/validator/semantic/) that validates every entry in `categories` and `policy_templates[].categories` against keys known from `pkg/categories`; register it in [`code/go/internal/validator/spec.go`](code/go/internal/validator/spec.go) at the appropriate minimum version(s).
5. Retrofit `validate_datastream_package_categories.go` from [#1095](https://github.com/elastic/package-spec/pull/1095): replace HTTP fetch with `pkg/categories`; remove URL constants and tests that require network access.

## Out of scope

- Adding a `subcategories` manifest field or restructuring how authors declare subcategories.
- Restricting `categories` to parent-only keys at a new `format_version` (version patches for enum shape).
- Changing how Kibana surfaces categories in the UI or the JSON shape of registry `/categories`.
- Hand-maintained enum ↔ YAML symmetry tests (Option B makes YAML authoritative; the semantic validator enforces allowed keys).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.