finos / finos/architecture-as-code

calm diff does not compare flows: deleting a flow reports no changes and --exit-code returns 0

Open
#2,998 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
399
Forks
138
Avg merge
2d 6h
Merged PRs (30d)
38

Description

`calm diff` compares nodes, relationships, control items and ADRs, but not `flows`. If the only difference between two architectures is in their flows (transitions removed, or a flow deleted completely) the diff comes back empty, and `--exit-code` returns 0. The help text suggests `--exit-code` for gating version bumps in CI, so a CI gate built on it will pass an architecture whose business processes have changed or disappeared.

Reproduced on 1.56.0, 1.55.0 and 1.53.0.

### Steps to reproduce

Save this as `a.json`. It validates cleanly on 1.56.0 (`hasErrors: false, hasWarnings: false`):

```json
{
"$schema": "https://calm.finos.org/release/1.0/meta/calm.json",
"nodes": [
{ "unique-id": "api", "node-type": "service", "name": "API", "description": "Receives requests." },
{ "unique-id": "db", "node-type": "database", "name": "DB", "description": "Stores records." },
{ "unique-id": "audit", "node-type": "service", "name": "Audit", "description": "Writes the audit trail." }
],
"relationships": [
{ "unique-id": "rel-api-db", "description": "API writes to DB.",
"relationship-type": { "connects": { "source": { "node": "api" }, "destination": { "node": "db" } } } },
{ "unique-id": "rel-api-audit", "description": "API records to audit.",
"relationship-type": { "connects": { "source": { "node": "api" }, "destination": { "node": "audit" } } } }
],
"flows": [
{ "unique-id": "flow-write", "name": "Write", "description": "A write with an audit record.",
"transitions": [
{ "relationship-unique-id": "rel-api-db", "sequence-number": 1, "description": "Persist." },
{ "relationship-unique-id": "rel-api-audit", "sequence-number": 2, "description": "Audit." }
] }
]
}
```

Drop a transition from the flow, so the audit step no longer happens:

```bash
jq '.flows[0].transitions |= .[0:1]' a.json > b.json
calm diff -a a.json -b b.json -f summary
calm diff -a a.json -b b.json --exit-code; echo "exit: $?"
```

Output on 1.56.0:

```
CALM architecture diff
----------------------
Nodes: +0 -0 ~0 ↔0 =3
Relationships: +0 -0 ~0 ↔0 =2
exit: 0
```

Same result if you delete the flow entirely:

```bash
jq '.flows = []' a.json > c.json
calm diff -a a.json -b c.json --exit-code; echo "exit: $?" # exit: 0
```

The JSON output tells the same story: with `-f json` the result object has 16 keys (nodesAdded/Removed/Modified/Same/Renamed, the edges equivalents, invalidItems, adrDiffItems, controlItems*) and the string "flow" appears nowhere in it.

### Expected

Flows compared like nodes and relationships are (flowsAdded/Removed/Modified, ideally with transition-level detail on modified), and flow changes counted by `--exit-code`. If that's out of scope for now, a note in the docs that `diff` skips flows would at least let CI consumers know to compare them separately.

### Context

We attach controls to flows (machine-readable test strategies) and gate releases on them. A flow-only edit is exactly the change that should invalidate an accepted control configuration, and right now it's invisible to `diff` — we compute a per-flow digest ourselves to work around it. Repointing a flow control's `config-url` is silent in the same way.

I see #2586 (control diffs) and #2526 (ADR diffs) are extending what `diff` covers, so maybe flows are already on someone's radar; couldn't find an issue for it.

Happy to put a PR together if the approach sounds reasonable — an initial version would mirror the existing element comparison over `flows[]`, matching flows by `unique-id` and transitions by `relationship-unique-id` plus `sequence-number`, feeding the same summary/JSON/exit-code paths.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the flow-only transition and deleted-flow cases with `calm diff`, including `--exit-code` and summary/JSON formats. Inspect the existing node and relationship comparison paths, then extend the same summary, JSON, and exit-code paths to `flows[]`, matching flows by `unique-id` and transitions by `relationship-unique-id` plus `sequence-number`. Done means flow changes are reported and make `--exit-code` nonzero.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.