galaxyproject / galaxyproject/foundry-pattern

Clarify validation layers and consider “Validate at every level possible”

Open
#41 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Astro
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Context

While reviewing the three **Build one** pages, the treatment of an “external check” turned out to be both more pervasive and less clearly modeled than expected:

- **The Model** places the external check outside the four structural parts.
- **Guiding Principles** folds it into “Deterministic Tools Do Deterministic Work.”
- **Anatomy of an Instance** treats the domain’s external check as a major extension-surface category.
- **Setting up a Foundry** makes building the external check a discrete implementation phase.
- **One Worked Example: the Astro Stack** explains why it has no fixed implementation row.

This may be pointing at a stronger and more general principle, tentatively:

> **Validate at every level possible.**

This issue is deliberately a design question, not a settled rewrite.

## Three validation levels to distinguish

### 1. Source validation

Does the knowledge base satisfy its authored contracts?

Examples:

- frontmatter and kind schemas
- resolvable references
- controlled vocabulary and registry membership
- license coherence
- generated catalogs and other self-documentation checks

### 2. Cast verification

Was the artifact built faithfully and reproducibly from the declared source?

Examples:

- deterministic output
- resolved reference and placement checks
- source and destination hashes
- provenance completeness
- drift detection and reproducible recasting

### 3. Domain evaluation

Does work performed with the artifact deserve trust in its domain?

Examples:

- Galaxy’s deterministic workflow validator
- statistical genomics’ empirical referee
- human review, proof tools, simulation, calibration, or negative controls

The first two appear intrinsic to the Foundry machinery. The third is domain-specific and may not admit one universal mechanism—but it may still express an important discipline: use the strongest independent validation the domain makes possible.

## Questions to resolve

- Should “Validate at every level possible” become a guiding principle?
- Which validation levels are requirements of the pattern, and which are instance-specific?
- Is domain evaluation part of the extension surface, or simply downstream domain machinery?
- How should we preserve “doing does not certify itself” without implying that every domain has a deterministic oracle?
- Is “external check” still useful terminology, or does it obscure the three distinct concerns?
- Where does validation end and evaluation begin?

## Likely documentation impact

Review and align:

- `content/pattern/the-model.md`
- `content/pattern/guiding-principles.md`
- `content/pattern/principles-in-action.md`
- `content/pattern/anatomy-of-an-instance.md`
- `content/pattern/setting-up-a-foundry.md`
- `content/pattern/standing-up-a-foundry.md`
- relevant case pages, especially the discussion of the pattern’s durable assets

## Possible direction—not yet a decision

- Treat source validation and cast verification as shared Foundry responsibilities.
- Treat domain evaluation as a domain-owned responsibility whose mechanism and attainable strength vary.
- Frame the principle as applying validation at every boundary where a meaningful independent check exists.
- Show the three levels explicitly instead of using “external check” as one catch-all term.

## Done when

- The three levels have clear names and boundaries.
- The Model and Guiding Principles agree about which guarantees belong to the pattern.
- The Build one pages distinguish shared validation machinery from domain-specific evaluation.
- Galaxy and statistical-genomics examples demonstrate the distinction without defining the abstraction by either implementation.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with content/pattern/the-model.md and content/pattern/guiding-principles.md, then compare the Build one pages listed in the issue. Trace how source validation, cast verification, and domain evaluation are currently described, including the Galaxy and statistical-genomics examples. Done means the three levels have clear boundaries, the pattern and examples agree, and shared machinery is distinguished from domain-owned evaluation.

Written by the indexing model from the issue text.

Assessment

Tech stack
markdown
Domain
documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.