finos / finos/architecture-as-code
Publish @finos/calm-models as the canonical CALM library + extract @finos/calm-io adapters
- Dominant language
- TypeScript
- Stars
- 399
- Forks
- 138
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 38
Description
## Feature Proposal
### Target Project:
`calm-models` (publishing + structure), with a new sibling package `calm-io`.
### Description of Feature:
**`@finos/calm-models` is treated across the platform as the canonical TypeScript CALM model — but it is not published, and that gap is already causing real divergence.** The package is `"private": true`; `npm install @finos/calm-models` returns a 404 today. It is consumable only as an in‑repo workspace symlink, not as a real dependency anyone outside the monorepo (or outside the npm workspace graph) can pull.
This matters because the rest of the platform already assumes it *is* a canonical, consumable library:
- **#2606** (Java `calm-models`) opens with "any user with a TS codebase can easily build integrations on top of `@finos/calm-models`" and proposes mirroring it for Java on Maven Central. That premise is not true today — there is no published TS artifact to build on, so the two sides start from an asymmetry.
- **CALM Studio**, contributed from the DTCC/FINOS hackathon, is the concrete cost of the gap. With no published canonical library to depend on, Studio carries **`@calmstudio/calm-core`** — its own Ajv‑based CALM model and validation layer — which has diverged from the canonical `@finos/calm-models` + Spectral approach. It is one of the **three parallel CALM type models/validators** that **#2600** enumerates (`@finos/calm-models` + Spectral in `shared`, Ajv in `@calmstudio/calm-core`, a hand‑rolled Zod model in CALM Guard), and it consumes `calm-models` only through a deep vendored `file:../../../../calm-models` path rather than a real package dependency. **#2600** has now been raised to consolidate these duplicated capabilities — a significant lift that exists, in part, because there was never a published canonical library to converge on in the first place.
So the underlying need is: **make `@finos/calm-models` the one published, canonical library that third parties — and our own surfaces like Studio and Guard — build on, instead of re‑implementing.** Publishing the package as‑is is not enough, though: to be the *single* library a consumer needs, it has to (a) be genuinely standalone (today it can't even resolve a `$ref` without reaching into `shared`), and (b) stay environment‑neutral so it's safe to depend on from a browser/edge/desktop context like Studio, not just Node tooling.
This proposal therefore couples **publishing** with a small **structural split** that makes `calm-models` cleanly publishable:
- **`@finos/calm-models` — canonical, environment‑neutral core (to be published).** Model, types, canonical, diff (as today) **plus** the `CalmReferenceResolver` contract, an `InMemoryResolver`, the `DereferencingVisitor` resolution algorithm, and a minimal injectable `Logger`. No filesystem, no network, no runtime‑specific globals — safe to depend on from Node, the browser, or a desktop app. No new runtime dependencies.
- **`@finos/calm-io` — optional environment adapters (to be published).** `FileReferenceResolver`, `HttpReferenceResolver`, `CompositeReferenceResolver`, `MappedReferenceResolver`. Depends only on `calm-models`. The package a Node consumer adds for ready‑made filesystem/HTTP `$ref` resolution.
- **`shared` — unchanged in role.** Keeps validation, docify, generate, document‑loaders, hub client, auth; consumes the resolvers from their new homes.
A third party (and Studio/Guard) then has one canonical library to align to — `calm-models` to parse/traverse/diff anywhere, `calm-io` when they want filesystem/HTTP resolution, or their own `CalmReferenceResolver` for a bespoke source — with no reason to fork a parallel model.
### User Stories:
- As a **maintainer**, I want `@finos/calm-models` published as the one canonical CALM type model, so that new surfaces (Studio, Guard, third‑party tools) consume it instead of forking a parallel model that later has to be reconciled (cf. #2600).
- As a **platform engineer building a browser/edge/desktop integration**, I want to depend on a published, environment‑neutral library to parse and traverse CALM, so that I don't pull Node‑only dependencies and don't re‑implement the model.
- As a **Node service author**, I want a ready‑made way to resolve `$ref`s from disk or over HTTP, so that I don't write my own resolver or depend on the CLI tooling package.
### Current Limitations:
- `@finos/calm-models` is `"private": true` and **unpublished** (npm 404). It cannot be `npm install`‑ed; only workspace‑internal consumers can use it.
- Because of that, **#2606's** core assumption (that the TS library is something users already build on) does not hold, and **CALM Studio** built a parallel model/validator (`@calmstudio/calm-core`, Ajv‑based) that diverged from the canonical model — now part of the **#2600** consolidation lift.
- Even if published as‑is, `calm-models` is **not standalone**: it defines the `Resolvable` contract but ships no resolver and no dereferencing walker, so it cannot resolve `$ref`s without `shared` — a heavyweight CLI/tooling package (Spectral, Handlebars, Axios, Winston, `calm-widgets`).
- The resolvers in `shared` are coupled to its Winston logger and read `process.env.DEBUG`, so they are not safe to ship in an environment‑neutral published core.
### Proposed Implementation:
**Publishing**
- Mark `@finos/calm-models` and `@finos/calm-io` publishable (`access: public`) and publish them to npm, so there is a real canonical artifact for third parties and for in‑repo surfaces (Studio/Guard) to converge on. (Timing/coordination with the #2600 consolidation and the #2606 Java publishing direction — see Open Questions.)
**Structural split (what makes it cleanly publishable)**
- Move into `calm-models` (environment‑neutral): the `CalmReferenceResolver` interface, `InMemoryResolver`, `CalmModelVisitor`, `DereferencingVisitor`, a small `getErrorMessage` util, and a minimal `Logger` interface with a `NoOpLogger` default.
- Create `calm-io` with the concrete adapters (`File`/`Http`/`Composite`/`Mapped`), depending only on `calm-models`.
- **Logger injection** replaces the static Winston logger: resolvers accept an optional `Logger` (default `NoOpLogger`), so the core/adapters carry no logging backend and read no `process.env` (mirrors the existing `setWidgetLogger` pattern in `calm-widgets`).
- `shared` repoints its imports (`template-processor`, `network-addressable-validator`, `logging-visitor`) and adds a `calm-io` dependency. **The CLI needs no changes** — it consumes everything through the `@finos/calm-shared` barrel.
**API / build**
- New `calm-models` subpath exports: `./resolver`, `./model-visitor`, `./logging`. No breaking changes to existing exports.
- `calm-io` slots into the workspace build order after `calm-models` and before `shared`. Its only new runtime dependency is `axios` (already centrally pinned via root `overrides`); the core gains no new dependencies.
- Consuming packages use `moduleResolution: "bundler"` (as `shared` already does) to read the `exports` map; cross‑package type re‑exports use `export type`.
A local proof‑of‑concept confirms the structural split is small and contained: ~5 files in `shared` repointed, the CLI untouched, and all affected TypeScript workspaces build and pass their test suites (`calm-models`, `calm-io`, `shared`, `cli`, the VSCode plugin).
### Alternatives Considered:
- **Publish `calm-models` as‑is, without the split.** Insufficient: it still can't resolve `$ref`s standalone, and it would publish `shared`'s Node/Winston coupling into a library meant to be environment‑neutral — undermining exactly the browser/desktop consumers (like Studio) whose divergence motivated this.
- **One package with a `@finos/calm-models/node` subpath instead of `calm-io`.** Rejected: a subpath export still lists Node‑only deps in the single package's `dependencies`, so a browser/edge consumer importing `/model` still drags `axios`/`fs` in. A separate package keeps the core's manifest clean.
- **Naming `calm-models-node`.** Rejected: `node` is a first‑class CALM concept (`CalmNode`). `calm-io` names the capability and stays accurate as more adapters are added.
- **Moving the document‑loaders into `calm-io` too.** Deferred — they're entangled with `shared`'s `SchemaDirectory`/`AuthPlugin`; larger change, can follow later.
- **Including validation in the published core.** Out of scope, consistent with #2606 deferring validation to the CALM server; validation stays in `shared`.
### Testing Strategy:
- Unit tests move with the code (`InMemoryResolver`/`DereferencingVisitor` → `calm-models`; `File`/`Http`/`Composite`/`Mapped` → `calm-io`, logger injected as a spy).
- New tests for the `Logger`/`NoOpLogger` contract and `getErrorMessage`.
- `shared`'s existing consumer tests run against the repointed imports; coverage thresholds maintained in every affected workspace.
### Documentation Requirements:
- A `calm-io` README (core‑vs‑adapter split, logger‑injection pattern).
- A `calm-models` docs note that resolution is now part of the canonical surface, with a "bring your own `CalmReferenceResolver`" example.
- A published‑package getting‑started snippet once publishing lands.
### Open Questions (I would like input from the other maintainers here):
1. **Publishing scope & timing.** Do we publish `calm-models` + `calm-io` now as the canonical artifacts, and how does that sequence against the #2600 consolidation and the #2606 Java/Maven Central direction?
- (a) Publish both now, as the foundation the #2600 convergence (Studio/Guard) aligns onto.
- (b) Land the structural split now; publish in a closely‑following follow‑up.
- (c) Hold publishing until #2600 has chosen the single model/validator.
2. **Versioning/stability.** What versioning + stability commitment do we make on a now‑public API (it has been free to churn as an internal workspace)?
3. **Document‑loaders.** Move them into `calm-io` eventually (relocating `SchemaDirectory`/`AuthPlugin`), or keep in `shared`?
4. **HTTP resolver.** Drop `axios` for global `fetch` (Node 22 + browsers) and graduate `HttpReferenceResolver` into the neutral core, leaving `calm-io` as essentially filesystem‑only — now or as a follow‑up? (Left in `calm-io` here to avoid changing an e2e test that asserts Axios's error‑message format.)
5. **Package name.** `calm-io` vs an alternative (`calm-resolvers`, `calm-models-fs`, …).
### Relationship to #2606 and #2600:
- **#2606** assumes a publicly consumable TS `calm-models`; this issue makes that real and keeps the TS/Java "one canonical model per platform" goal symmetric, with the same core‑vs‑adapter boundary mapping onto the Java artifacts.
- **#2600** is consolidating the parallel models/validators (Studio's `@calmstudio/calm-core`, Guard's Zod model) onto the canonical one. A published, standalone, environment‑neutral `calm-models` is the artifact that convergence aligns onto — and the structural prerequisite that stops the next Studio‑style divergence.
### Implementation Checklist:
- [ ] Design reviewed and approved
- [ ] Publishing scope & timing agreed (see Open Questions)
- [ ] Implementation completed
- [ ] Tests written and passing
- [ ] Documentation updated
- [ ] Relevant workflows updated (publish pipeline for `calm-models`/`calm-io`)
Contributor guide
Assessment
This issue has not been assessed yet.