finos / finos/architecture-as-code

Feature Proposal: find all architectures that implement a given pattern (reverse pattern→architecture lookup)

Open
#2,918 1 comment 0 reactions 1 assignee Claimed by @YoofiTT96 View on GitHub
enhancement needs-input
Dominant language
TypeScript
Stars
399
Forks
138
Avg merge
2d 14h
Merged PRs (30d)
37

Description

## Feature Proposal

### Target Project:
`calm-hub` (Java/Quarkus backend), with supporting changes likely in `shared` (generate/instantiate flow) and `calm-hub-ui`.

### Description of Feature:
Provide a way to ask CALM Hub: **"which architectures implement pattern Y?"** — i.e. given a stored pattern, return the set of architectures that were derived from / conform to it.

This is a reverse-lookup of the pattern→architecture relationship. It's valuable for governance and impact analysis: when a pattern changes (e.g. a new control requirement is added), maintainers need to know which downstream architectures are affected; reviewers want to confirm an architecture actually derives from an approved pattern; and consumers want to discover reference implementations of a pattern.

> **Note:** This issue is framed as an **investigation/spike** as much as a feature. The current data model does not store this relationship in any queryable form, so the first deliverable is a decision on *how* to represent and query the linkage — not just an endpoint over existing data.

### User Stories:
- As a **pattern maintainer**, I want to list all architectures implementing a pattern, so that I can assess the blast radius before changing or deprecating that pattern.
- As an **architecture reviewer**, I want to confirm an architecture derives from an approved pattern, so that I can enforce governance.
- As an **architect**, I want to discover existing implementations of a pattern, so that I can reuse them as references.

### Current Limitations:
There is **no stored, queryable link** from an architecture back to its source pattern today.

- **Domain models carry no reference.** `domain/Architecture.java` stores only `namespace`, `name`, `description`, `id`, `version`, and the raw `architecture` JSON string. `domain/Pattern.java` is symmetric. Neither has a `patternId` / `derivedFrom` / `$ref` field. The write DTO `domain/architecture/ArchitectureRequest.java` accepts only `name`, `description`, `architectureJson` — no pattern reference on write.
- **The only existing trace is a soft, by-convention URL inside the JSON blob.** When an architecture is generated from a pattern, `shared/src/commands/generate/components/instantiate.ts` (~L160) sets the output document's `$schema` to the pattern's `$id`:
```js
if (pattern.$id) {
output.$schema = pattern.$id; // architecture's $schema = pattern's $id URL
}
```
This is fragile as a linkage: (a) it's inside the opaque JSON blob the hub stores verbatim (`versions: {"1-0-0": }`), not a first-class field; (b) it's a **URL string** (e.g. `.../options-prototype.pattern.json`) that does **not** map to the hub's internal `(namespace, patternId, version)` identity; (c) it's only set by the CLI/`shared` generate path — architectures created directly via the REST API set whatever `$schema` the caller supplies, and many examples point `$schema` at the meta-schema (`.../meta/calm.json`) instead of a pattern; so it is inconsistent and not guaranteed present.
- **No store method or index supports the query.** `store/ArchitectureStore.java` and `PatternStore.java` only expose namespace/id/version lookups. There is no `getArchitecturesImplementingPattern` and no field/content index.
- **Global Search doesn't help.** `resources/SearchResource.java` → `store/mongo/MongoSearchStore.java` does a case-insensitive substring match against only the `name` and `description` fields (`store/util/SearchTextMatcher`), and **never reads the architecture JSON body**. So it cannot match on `$schema` today either.

**Does the current model support this? No — not as a query.** The relationship is either absent or only recoverable heuristically from an unindexed, inconsistent field buried in the document. Delivering the feature requires either reworking how the pattern reference is captured/persisted, or accepting a best-effort scan of the existing `$schema`.

### Proposed Implementation:
Three candidate approaches to evaluate, presented without a preferred path — the choice is the main output of this investigation.

---

#### Option A — Best-effort `$schema` scan (low effort, medium precision)
Read the `$schema` value out of each stored architecture blob and match it against the target pattern.

- **Model impact:** none — no schema change; relies on the existing `$schema`/`$id` convention.
- **Query:** new `SearchStore`/store method that, for a given pattern, iterates namespace architecture documents and matches `$schema` (needs a URL↔hub-pattern resolution strategy — e.g. match on the pattern's own `$id`, or on a canonical hub URL for `(namespace, patternId, version)`).
- **API:** e.g. `GET /calm/namespaces/{ns}/patterns/{patternId}/versions/{version}/implementations` returning matching architecture summaries.
- **Trade-offs:** cheap and non-invasive, ships fast. But only finds architectures produced by the generate flow, misses API-created ones, and the URL→identity mapping is brittle. Precision depends entirely on convention adherence.

#### Option B — First-class traceability metadata (high effort, high precision)
Persist a structured, indexed pattern reference on the architecture.

- **Model impact (rework):** add an optional `sourcePattern` reference (`namespace`, `patternId`, `version`) to `Architecture` / `ArchitectureRequest` and to the stored document; populate it on create (both REST and the `shared` generate path, resolving `$id`→hub identity there); index it (Mongo index; Nitrite equivalent).
- **Migration:** backfill existing architectures by best-effort parsing their `$schema` (i.e. reuse Option A's scan once to seed the new field). Handle documents with no resolvable pattern.
- **API:** clean reverse-lookup endpoint; also enables showing "derived from pattern X" on an architecture.
- **Trade-offs:** the durable, correct answer — precise, indexable, works regardless of creation path. But touches the domain model, both store backends, the write path, the generate flow, and needs a migration/backfill. Higher long-term surface area to maintain, but the relationship becomes explicit and trustworthy.

#### Option C — Structural heuristic matching (medium effort, low/fuzzy precision)
Infer "implements" by comparing structure (node-types, controls, relationships) between architecture and pattern.

- **Model impact:** none.
- **Trade-offs:** works even without any pattern reference, but "implements" becomes a similarity judgement with false positives/negatives and no clear threshold. Best treated as a future enhancement, not the primary mechanism.

---

#### Long-term maintenance considerations
- **A** adds little maintenance burden but leaves a permanently unreliable answer, and every future "why is this pattern missing implementations?" question re-surfaces the convention gap.
- **B** is more to build and maintain (two store backends, migration, keeping the reference in sync on updates), but converts an implicit convention into an explicit, testable contract — cheaper to reason about over time and unblocks related features (impact analysis, "derived from" UI).
- Either way, the **`$id`/`$schema` convention should be documented and, ideally, validated** so the linkage stops being accidental. Consider whether the hub should reject or warn on architectures whose `$schema` claims a pattern that doesn't resolve.
- Cross-namespace and versioning semantics need a decision: does "implements pattern Y" pin to a specific pattern version, or any version of that pattern id?

### Alternatives Considered:
- **Extending global Search to read the JSON body** and match `$schema`: reuses existing infrastructure but bloats a substring-search feature with structured-relationship semantics; better as a dedicated query (still effectively Option A under the hood).
- **Computing the relationship purely client-side in `calm-hub-ui`**: pushes blob-parsing to the browser and doesn't scale; rejected in favour of a server-side store method.

### Testing Strategy:
- **Unit:** store method resolves matches for the `$schema`/`$id` convention; handles missing/non-pattern `$schema`, meta-schema URLs, and non-resolving URLs gracefully. Both Mongo and Nitrite store implementations.
- **Integration (TestContainers):** seed patterns + architectures (generated and API-created), call the new endpoint, assert correct grouping/pagination and namespace-permission filtering (mirroring `SearchResource` readable-namespace scoping).
- **Regression:** for Option B, a migration/backfill test over a fixture with mixed architectures.
- **shared/CLI:** if the generate path is updated to emit a hub-resolvable reference, add tests in `shared` around `instantiate.ts`.

### Documentation Requirements:
- Document the `$schema` = pattern `$id` convention and how the hub interprets it (calm-hub docs + user docs on calm.finos.org).
- OpenAPI/Swagger for the new endpoint.
- If Option B: document the new `sourcePattern` field and migration behaviour.

### Implementation Checklist:
- [ ] Design reviewed and approved (which option, versioning semantics)
- [ ] Implementation completed
- [ ] Tests written and passing (unit + integration, both store backends)
- [ ] Documentation updated
- [ ] Relevant workflows updated (if needed)
- [ ] Performance impact assessed (scan cost vs. indexed lookup)

### Additional Context:
This proposal is deliberately scoped as an investigation first: the key open question is **whether to keep relying on the by-convention `$schema` URL (Option A), promote the pattern linkage to first-class indexed metadata (Option B), or infer it structurally (Option C)** — the trade-offs above are for maintainers to weigh, not pre-decided here. The URL↔hub-identity mapping (`$id` URL vs. internal `(namespace, patternId, version)`) is the crux for A and B either way and should be resolved before implementation.

Relevant code:
- `calm-hub/src/main/java/org/finos/calm/domain/Architecture.java`, `Pattern.java`, `architecture/ArchitectureRequest.java`
- `calm-hub/src/main/java/org/finos/calm/store/{ArchitectureStore,PatternStore,SearchStore}.java` and `store/mongo`, `store/nitrite` impls
- `calm-hub/src/main/java/org/finos/calm/resources/{ArchitectureResource,PatternResource,SearchResource}.java`
- `shared/src/commands/generate/components/instantiate.ts` (`$schema = pattern.$id`)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.