galaxyproject / galaxyproject/foundry-pattern
Clarify validation layers and consider “Validate at every level possible”
- Dominant language
- Astro
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Context
While reviewing the three **Build one** pages, the treatment of an “external check” turned out to be both more pervasive and less clearly modeled than expected:
- **The Model** places the external check outside the four structural parts.
- **Guiding Principles** folds it into “Deterministic Tools Do Deterministic Work.”
- **Anatomy of an Instance** treats the domain’s external check as a major extension-surface category.
- **Setting up a Foundry** makes building the external check a discrete implementation phase.
- **One Worked Example: the Astro Stack** explains why it has no fixed implementation row.
This may be pointing at a stronger and more general principle, tentatively:
> **Validate at every level possible.**
This issue is deliberately a design question, not a settled rewrite.
## Three validation levels to distinguish
### 1. Source validation
Does the knowledge base satisfy its authored contracts?
Examples:
- frontmatter and kind schemas
- resolvable references
- controlled vocabulary and registry membership
- license coherence
- generated catalogs and other self-documentation checks
### 2. Cast verification
Was the artifact built faithfully and reproducibly from the declared source?
Examples:
- deterministic output
- resolved reference and placement checks
- source and destination hashes
- provenance completeness
- drift detection and reproducible recasting
### 3. Domain evaluation
Does work performed with the artifact deserve trust in its domain?
Examples:
- Galaxy’s deterministic workflow validator
- statistical genomics’ empirical referee
- human review, proof tools, simulation, calibration, or negative controls
The first two appear intrinsic to the Foundry machinery. The third is domain-specific and may not admit one universal mechanism—but it may still express an important discipline: use the strongest independent validation the domain makes possible.
## Questions to resolve
- Should “Validate at every level possible” become a guiding principle?
- Which validation levels are requirements of the pattern, and which are instance-specific?
- Is domain evaluation part of the extension surface, or simply downstream domain machinery?
- How should we preserve “doing does not certify itself” without implying that every domain has a deterministic oracle?
- Is “external check” still useful terminology, or does it obscure the three distinct concerns?
- Where does validation end and evaluation begin?
## Likely documentation impact
Review and align:
- `content/pattern/the-model.md`
- `content/pattern/guiding-principles.md`
- `content/pattern/principles-in-action.md`
- `content/pattern/anatomy-of-an-instance.md`
- `content/pattern/setting-up-a-foundry.md`
- `content/pattern/standing-up-a-foundry.md`
- relevant case pages, especially the discussion of the pattern’s durable assets
## Possible direction—not yet a decision
- Treat source validation and cast verification as shared Foundry responsibilities.
- Treat domain evaluation as a domain-owned responsibility whose mechanism and attainable strength vary.
- Frame the principle as applying validation at every boundary where a meaningful independent check exists.
- Show the three levels explicitly instead of using “external check” as one catch-all term.
## Done when
- The three levels have clear names and boundaries.
- The Model and Guiding Principles agree about which guarantees belong to the pattern.
- The Build one pages distinguish shared validation machinery from domain-specific evaluation.
- Galaxy and statistical-genomics examples demonstrate the distinction without defining the abstraction by either implementation.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with content/pattern/the-model.md and content/pattern/guiding-principles.md, then compare the Build one pages listed in the issue. Trace how source validation, cast verification, and domain evaluation are currently described, including the Galaxy and statistical-genomics examples. Done means the three levels have clear boundaries, the pattern and examples agree, and shared machinery is distinguished from domain-owned evaluation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- markdown
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100