finos / finos/architecture-as-code
Calm Hub: Introduce NodeStore Interface
- Dominant language
- TypeScript
- Stars
- 399
- Forks
- 138
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 38
Description
## Feature Proposal
### Target Project:
`calm-hub` — a new first-class store, served through the existing namespace-scoped REST API. A downstream consumption path (CLI / `calm-models` / VS Code plugin resolving nodes from CALM Hub) is in scope as a follow-on but is not the primary deliverable.
### Description of Feature:
Introduce a **NodeStore** in CALM Hub alongside the existing Controls, ADRs, Standards, Flows, Patterns, Decorators, and Interfaces stores. It provides a namespace-scoped, versioned store for reusable **custom node definitions**, so an organization can publish a private, concrete catalog of node types and distribute it to all consumers through CALM Hub.
Today, a genuinely new node type must be bundled into the consuming tooling (for example, the VS Code plugin). That means every catalog change forces a tooling release and redistribution. Serving nodes centrally removes that coupling: consumers pull the approved catalog from CALM Hub, and node changes do not require continuous plugin updates.
Like the other stores, the NodeStore supports all three CALM Hub backends — MongoDB, NitriteDB, and the read-only GitHub backend. For GitHub-backed namespaces, node definitions are sourced straight from a Git repository by scanning predefined folder names recursively, so a team can manage its node catalog as files in Git and have CALM Hub serve them without a separate publish step.
### User Stories:
- As a **Enterprise Architecture team**, I want to publish a private set of custom node definitions to CALM Hub so that all consumers use an approved catalog without waiting for a tooling release.
- As an **architect (consumer)**, I want to retrieve organization-specific node definitions from CALM Hub so that I can compose architectures from approved building blocks.
- As a **CALM Hub admin**, I want node definitions to be namespace-scoped and permission-controlled so that catalogs stay private to the right teams.
- As a **tooling maintainer**, I want nodes served centrally so that I do not ship a plugin update every time a node is added or changed.
- As a **team using a GitHub-backed namespace**, I want CALM Hub to source node definitions from predefined folders in our repository, scanned recursively, so that our catalog is managed as code in Git and served without a separate publish step.
### Current Limitations:
- **Standards** extend core CALM definitions through `allOf` composition (for example, requiring `costCenter` and `owner` on all nodes). They add constraints and required properties to _existing_ nodes; they cannot introduce entirely new node types.
- **Pattern + Standard** layered governance covers "what nodes must exist" + "what properties they must have," but both assume the node types already exist in core CALM or in bundled tooling.
- Custom node definitions must therefore be distributed inside consuming tools, so the catalog cannot change without a release and redistribution.
- CALM Hub already centralizes Architectures, Patterns, Controls, ADRs, Standards, Flows, Decorators, Interfaces, and Domains — but has no equivalent store for reusable node definitions.
- The GitHub-backed backend discovers resources by folder-name convention (`CalmContentDetector` — `architectures/`, `standards/`, `controls/`, …) classified through `CalmResourceType`. Neither a `NODE` type nor a node folder convention exists today, so node files in a repository are ignored.
### Proposed Implementation:
Mirror the existing store types (`StandardStore` / `ControlStore` / `AdrStore`) so the NodeStore is consistent with the current architecture. New components:
| Layer | Component | Notes |
| --------- | ------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Interface | `NodeStore` (`org.finos.calm.store`) | Namespace-scoped, versioned CRUD, matching the Standard/Control shape |
| Storage | `MongoNodeStore`, `NitriteNodeStore`, `GitHubNodeStore` | Implementations under `store/mongo/`, `store/nitrite/`, `store/github/` |
| Selection | `NodeStoreProducer` | CDI producer selecting mongo / standalone / github on `calm.database.mode` |
| REST | `NodeResource` | `/calm/namespaces/{namespace}/nodes`, guarded by `@PermissionsAllowed(READ/WRITE)` |
| Domain | `Node`, `CreateNodeRequest`, node exception types | Node exceptions mirror the Standard/ADR set |
- **Technical design:** Namespace-scoped and versioned (list nodes, create node, list versions, get version, create version, delete), following the ID + version model already used by Standards.
- **API changes:** New GET/POST endpoints only; no change to existing routes. All endpoints declare `@Produces(APPLICATION_JSON)` and sanitize any echoed user input with `STRICT_SANITIZATION_POLICY` per the repo XSS rules.
- **Data model changes:** New collection/repository keyed by `namespace + nodeId + version` holding the node JSON definition in MongoDB and NitriteDB (including read-only mode); the GitHub backend uses git SHAs as versions.
- **Dependencies on other components:** Optional follow-ons — Search integration so nodes are discoverable, an MCP tool for node lookup, and a consumer-side resolver so a namespace-qualified node reference can point at a CALM Hub node.
**GitHub-backed sourcing (recursive folders):**
- Add `NODE` to `CalmResourceType` and register `nodes/` as a known folder in `CalmContentDetector` (alongside the existing `architectures/`, `standards/`, … conventions). The detector already walks every path segment (`getParentDirectoryName`), so node files are discovered recursively at any depth under a `nodes/` folder.
- `GitHubNodeStore` reads from the per-namespace clone, indexes node files via `InMemoryRegistryService`, and versions by git SHA through `GitHubVersionService`.
- Writes are unsupported in GitHub mode (`GitHubWriteNotSupportedException`), matching the other GitHub stores; the repository stays the source of truth.
### Alternatives Considered:
| Alternative | Why rejected |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| Standards (`allOf` composition) | Adds required properties to existing nodes; cannot define new node types |
| Patterns | Describe required node instances/topology, not a reusable catalog of node types |
| Bundling nodes in the VS Code plugin / tooling | Requires continuous releases and redistribution; no central governance or per-namespace privacy |
| Detecting nodes in Git by JSON structure only (no folder convention) | Rejected — a folder-name convention reuses `CalmContentDetector` and gives predictable, recursive discovery |
### Testing Strategy:
- **Unit (JUnit 5):** `NodeStore` implementations and `NodeResource`, covering success and error cases (>80% on new code per repo policy).
- **Integration (TestContainers):** MongoDB path; NitriteDB standalone and read-only paths (writes rejected in read-only).
- **Security:** per-namespace READ/WRITE enforcement and cross-namespace isolation.
- **XSS:** error responses that echo user input are sanitized.
- **Smoke tests:** extend `smoke-test.sh` (readonly + readwrite) to cover the new endpoints.
- **GitHub backend:** recursive discovery under the predefined node folders, registry indexing, git-SHA version listing/fetch, and write rejection in GitHub mode.
### Documentation Requirements:
- CALM Hub developer guide "Store Interfaces" section (https://calm.finos.org/calm-hub/calm-hub-developer-guide/#the-store-interfaces).
- OpenAPI/Swagger annotations for the new endpoints.
- `calm-hub/AGENTS.md` store and resource lists.
- `calm-hub/PERMISSIONS.md` if any scope/grant changes are needed.
- Consumer-facing guidance on publishing and consuming nodes from CALM Hub.
- GitHub backend setup: enabling `calm.database.mode=github`, the `nodes/` folder convention, and recursive discovery behavior.
### Implementation Checklist:
- [ ] Design reviewed and approved
- [ ] Implementation completed
- [ ] Tests written and passing
- [ ] Documentation updated
- [ ] Relevant workflows updated (if needed)
- [ ] Performance impact assessed
### Additional Context:
Raised in office hours with @jpgough-ms and @rocketstack-matt. The concrete node set is private to our organization, so this proposal asks for the distribution _mechanism_, not the specific nodes.
```mermaid
flowchart LR
Repo["GitHub repo — nodes/ folders"] -->|recursive sync| Hub[(CALM Hub NodeStore)]
Team["Enterprise Architecture team"] -->|publish versioned nodes| Hub
Hub -->|resolve on demand| CLI[CALM CLI]
Hub -->|resolve on demand| Plugin[VS Code Plugin]
Hub -->|resolve on demand| Consumers[Other Consumers]
```
Contributor guide
Research direction
Compare StandardStore, ControlStore, and AdrStore first to map the NodeStore interface, implementations, producer, domain types, and REST resource. Then trace CalmResourceType and CalmContentDetector for recursive GitHub discovery, and review the stated JUnit 5, TestContainers, security, XSS, and smoke-test coverage. Done means the three backends, namespace/versioned endpoints, GitHub read-only behavior, tests, and listed documentation are complete and the design is approved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, java, mongodb
- Domain
- api, backend, databases, documentation, security, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100