elastic / elastic/package-spec
Migrate mode-dependent field and manifest constraints into the spec
- Dominant language
- Go
- Stars
- 20
- Forks
- 93
- Avg merge
- 15h 10m
- Merged PRs (30d)
- 12
Description
## Context
[#549](https://github.com/elastic/package-spec/issues/549) introduced source and build validation modes and explicitly deferred two workstreams as out-of-scope follow-ups:
- **Migrating semantic validators to the spec** — source/build constraints currently enforced as Go validators rather than spec schema rules.
- **JSON schema migration** — encoding source/build expectations directly in the JSON schema (tracked in [elastic/ingest-dev#7145](https://github.com/elastic/ingest-dev/issues/7145)).
This issue picks up the first of those. The implementation in [PR #1178](https://github.com/elastic/package-spec/pull/1178) resolved two of the four acceptance criteria from #549 via semantic validators:
- **AC#3** — build mode rejects fields with `external: ecs` references (`validate_no_external_fields.go`)
- **AC#4** — build mode rejects data stream manifest streams carrying a `package:` field (`validate_stream_input_bundled.go`)
The analogous constraints for source-only folders and file extensions (`_dev/`, `.link`) were expressed at the spec level in that same PR as prior art for the folder/file case. The question is whether the field-content and manifest-content constraints can follow the same direction.
## Problem
The current setup means the spec schema alone cannot determine validity — you also need to know which semantic validators were applied and in which mode. Ideally, a mode-aware spec would encode these constraints so schema validation covers them without additional Go code.
The core obstacle is that `FileSchemaLoadOptions` has no concept of mode. JSON Schema files are loaded once without mode context, so there is currently no mechanism to enforce "property X is forbidden only in built mode" at the schema level.
## Open Question
How should we approach making field and manifest schemas mode-aware? Some directions worth discussing:
1. **Mode annotation on spec properties** — introduce an annotation (e.g. `x-source-only: true` or a `forbiddenIn: [build]` property) on JSON Schema fields in the spec files. A mode-aware loader would read the annotation and apply additional `not:` constraints when validating in the relevant mode. This is the direction hinted at by OQ#4 in #549 ("design a generic source/build/both marker for spec entries") and relates to OQ#3 ("finish `visibility: private`"). It would keep constraints co-located with the field definitions without duplicating spec files.
2. **Dual spec files per mode** — maintain separate spec variants (e.g. `fields.spec.yml` / `fields.built.spec.yml`) with `FileSchemaLoadOptions` selecting the file based on mode. Explicit but requires keeping two files in sync.
3. **Mode field in schema load options with conditional `$ref`** — add a `Mode` field to `FileSchemaLoadOptions` and use it to resolve `$ref`s to stricter schema variants for built mode, similar to how `if/then/else` is already used in the existing spec files.
4. **Keep semantic validators** — the current approach is pragmatic and self-contained. The cost of mode-conditional schema loading may not be justified unless there is a broader initiative (see [ingest-dev#7145](https://github.com/elastic/ingest-dev/issues/7145)).
## What we'd like to understand
- Is there appetite to invest in mode-aware schema loading, separate from or coordinated with [ingest-dev#7145](https://github.com/elastic/ingest-dev/issues/7145)?
- Which of the approaches above best fits the existing architecture?
- Should this be scoped to just these two validators, or designed as a general mechanism for future mode-dependent constraints (resolving OQ#4 from #549)?
## References
- [#549 — Add different validation modes for source and built packages](https://github.com/elastic/package-spec/issues/549)
- [elastic/ingest-dev#7145 — JSON schema migration](https://github.com/elastic/ingest-dev/issues/7145)
- [PR #1178 — Semantic validation for build mode](https://github.com/elastic/package-spec/pull/1178)
Contributor guide
Research direction
Review validate_no_external_fields.go and validate_stream_input_bundled.go alongside the source-only folder and file-extension constraints from PR #1178. Then inspect FileSchemaLoadOptions and the existing if/then/else usage in the spec files. Done means agreeing on a general mode-aware schema approach and defining its implementation scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100