finos / finos/architecture-as-code
GitHub as a backing store for CALM
- Dominant language
- TypeScript
- Stars
- 399
- Forks
- 138
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 37
Description
## Feature Proposal
### Target Project:
Open question — this proposal identifies two possible target projects (`calm-hub` or `cli`) rather than assuming one. See "Proposed Implementation" below; I would like input from the other maintainers here on which is the better fit, or whether both are worth pursuing.
### Description of Feature:
Allow a GitHub repository to serve as a persistence backend for CALM documents (namespaces, architectures, patterns, flows, etc.), so a team can get PR review, diffs, and git history "for free" without operating a database. The core motivation is a **zero-ops** option: a GitHub repo + a PAT as the entire backing store, aimed at small teams, demos, and GitOps-oriented shops.
### User Stories:
- As a small team evaluating CALM, I want to store our architecture documents without standing up MongoDB or managing a NitriteDB data volume, so that adoption has near-zero infrastructure cost.
- As an architecture reviewer, I want changes to CALM documents to go through the same PR review my team already uses for code, so that architecture changes get the same scrutiny.
- As a CalmHub operator who already runs the server, I want an alternative to Mongo/Nitrite that stores data in a repo I control, so that my system of record is plain git rather than a database I have to back up and administer.
### Current Limitations:
- `calm-hub`'s storage layer is pluggable (`org.finos.calm.store.*`) but only has `mongo` and `standalone` (embedded NitriteDB) modes today — both require running the Quarkus server, and neither gives git-native history/PR review over the data itself.
- The CLI's `calm workspace` command is already a local, git-rooted bundle of CALM documents, but its only remote sync target is CalmHub via `push`/`check`/`bump` (one-way, no `pull` yet — tracked as TODO from #2378). It has no notion of syncing directly to an arbitrary GitHub repo.
### Proposed Implementation:
Two structurally different connection points were scoped out; they solve related but distinct problems and are presented here as alternatives rather than a single recommendation.
**Option A — CalmHub storage backend.** Add a `github` mode to calm-hub's pluggable store layer, alongside `mongo`/`standalone`, following the existing "Adding a New Storage Backend" runbook in `calm-hub/AGENTS.md`.
- New package `org.finos.calm.store.github` implementing the 13 "content" store interfaces (`NamespaceStore`, `ArchitectureStore`, `PatternStore`, `FlowStore`, `AdrStore`, `ControlStore`, `DomainStore`, `InterfaceStore`, `CoreSchemaStore`, `DecoratorStore`, `StandardStore`, `TimelineStore`, `ResourceMappingStore`) plus a `GithubCounterStore` for ID generation. `UserAccessStore`, `AuditLogStore`, `SearchStore` are explicitly out of scope for phase 1 (see below).
- One configured repo per Hub instance; namespaces as top-level folders; one file per version (`{namespace}/{resourceType}/{id}/{version}.json`), mapping 1:1 onto the existing Mongo versions-map.
- Direct commits to a configured branch (synchronous, matching existing Mongo/Nitrite write semantics). The one-file-per-version layout means GitHub's own Contents API optimistic-concurrency check (`sha` matching, create-only PUTs) *is* the conflict detector — no custom locking needed. New resource IDs use a best-effort max+1-and-retry-on-409 scheme (not a hard atomic guarantee like Mongo's counter).
- In-memory cache with TTL over tree listings/file contents; versions are immutable once written so per-version content is cacheable indefinitely.
- A thin hand-written `@RegisterRestClient` interface for the specific GitHub REST endpoints needed (rather than a full GitHub SDK object model), matching the codebase's existing small-mockable-interface test style.
- Auth via a fine-grained PAT scoped to the one configured repo.
- Pro: every existing Hub consumer (calm-hub-ui, VSCode extension, calm-studio, `workspace push`) keeps working unchanged. Con: doesn't remove the operational footprint of running the Quarkus server — only the database goes away, so "zero-ops" is partial.
**Option B — CLI `workspace` push/pull to GitHub.** Add GitHub as a second sync target for `calm workspace`, alongside CalmHub, with no server involved at all.
- Introduce a `RemoteStore`-style abstraction (mirroring calm-hub's own pluggable-store pattern, but at the CLI layer): a `CalmHubRemoteStore` (the existing `CalmHubClient` behavior, refactored behind the interface) and a new `GithubRemoteStore`.
- `.calm-workspace/config.json` gains a remote target config (repo owner/name, branch, PAT via env var, namespace→path prefix). `push` commits tracked documents using the same one-file-per-version layout as Option A, via the GitHub Contents API.
- Requires building `pull` (reading the repo structure back and materializing local bundle files), which doesn't exist today for *any* remote target — this is genuinely new scope, not a side effect of adding a backing store.
- Pro: genuinely zero-server, best fit for the stated zero-ops motivation; smaller conceptual leap since a workspace is already git-rooted. Con: only serves the CLI — calm-hub-ui/VSCode/Studio get nothing from this unless they grow their own GitHub client, or CalmHub is later taught to read from a workspace-populated repo (a possible future convergence of both options, not designed here).
### Alternatives Considered:
The two options above are the alternatives; neither is recommended over the other in this proposal. A third possibility — CalmHub reading from a GitHub repo that a workspace populates, effectively layering Option A on top of Option B's data — was noted as a possible future convergence but is out of scope for an initial implementation.
### Testing Strategy:
- **Option A:** Unit tests mock the thin GitHub REST client interface directly (consistent with existing Mongo/Nitrite unit tests, which mock typed interfaces rather than using `@SuppressWarnings("unchecked")`). Integration tests (`GithubArchitectureIntegration.java` etc., mirroring the existing `Mongo*Integration`/`Nitrite*Integration` naming pattern) run against a local WireMock instance with scripted GitHub API responses — no TestContainers image exists for GitHub, and this avoids live network dependency or CI secrets.
- **Option B:** Unit tests around the new `RemoteStore` abstraction and `GithubRemoteStore`, likely also backed by a mocked/stubbed GitHub client; `push`/`pull` round-trip tests against a local git repo or stubbed API responses.
### Documentation Requirements:
- `calm-hub/AGENTS.md` (Option A) or `cli/AGENTS.md` (Option B) updated with the new mode/target, config properties, and known limitations.
- User-facing docs on calm.finos.org describing setup (PAT scope, repo layout expectations).
- Explicit documentation of the open questions below as known gaps, not silent omissions.
### Implementation Checklist:
- [ ] Design reviewed and approved (including which option — A, B, or both — is chosen)
- [ ] Implementation completed
- [ ] Tests written and passing
- [ ] Documentation updated
- [ ] Relevant workflows updated (if needed)
- [ ] Performance impact assessed
### Additional Context:
Open questions raised by this proposal that need resolution regardless of which option is chosen:
- **UserAccessStore** (Option A only): could map to GitHub collaborator/team permissions, but Hub's namespace-scoped role model doesn't line up cleanly with GitHub's repo-level permission model.
- **AuditLogStore** (Option A only): git commit history is a natural fit for "what changed when," but commits would be authored by the Hub's service PAT, not the actual end user — actor attribution is lost unless Hub embeds user identity in commit metadata/trailers.
- **SearchStore** (Option A only): no query engine — would need an in-memory index built from cached repo contents; full-text semantics from Mongo/Nitrite likely won't translate directly.
- **Rate limits at scale:** GitHub's 5000 req/hr (PAT-authenticated) is shared across all callers using that token — more of a concern for Option A (many concurrent Hub users through one server) than Option B (single CLI invocation at a time).
Contributor guide
Research direction
First resolve whether the work targets calm-hub, cli, or both, then read calm-hub/AGENTS.md or cli/AGENTS.md and the existing pluggable storage or workspace push implementation. For Option A, inspect the existing Mongo/Nitrite integrations and the proposed GithubArchitectureIntegration.java pattern; for Option B, inspect the CalmHubClient and workspace commands. Done means the selected scope is implemented, tested with mocked or local API responses, and documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, github, mongodb, typescript
- Domain
- api, backend, cli, database, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100