Add Field-Level Data Documentation and Movement Tracking to CALM
- Dominant language
- JavaScript
- Stars
- 2
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Description:
As CALM users increasingly leverage the framework to define complex systems, there is a growing need to explicitly document what data exists, how it moves across the architecture, and what controls or transformations are applied. Currently, CALM provides limited mechanisms for specifying detailed data contents or movement semantics, which leads to gaps in clarity and compliance.
This proposal introduces functionality allowing architects and developers to explicitly document data at a field level, describe its movement (real-time, streaming, batch), lineage across the system, and integrate controls directly into the architecture definition.
## Proposed Feature:
Introduce a structured way within CALM to:
1. Define data elements clearly with names, types, and descriptions (e.g., customer_address: String - Primary shipping address of the customer).
2. Specify data movement modes (e.g., real-time, batch, streaming) and document the details of interactions between architectural components.
3. Integrate data lineage documentation to track the origins, transformations, and destinations of data.
4. Annotate data elements with metadata, including classifications (e.g., PII, confidential), governance controls, and source-of-truth references.
5. Visualise data flows, lineage, and metadata directly within CalmHub, enhancing operational clarity.
Implementation Using CALM Core Concepts:
- Nodes represent architectural components that produce or consume data.
- Interfaces specify explicit communication protocols (e.g., HTTP, Kafka, SQL).
- Relationships document interactions, enhanced to include detailed data elements and movement semantics:
```json
{
"relationship-type": {
"interacts": {
"source": {"node": "OrderService", "interfaces": ["REST-API"]},
"destination": {"node": "BillingService", "interfaces": ["REST-Endpoint"]}
}
},
"protocol": "HTTPS",
"mode": "real-time",
"data": [
{
"name": "order_id",
"type": "UUID",
"description": "Unique identifier for the order",
"classification": "Internal"
},
{
"name": "customer_address",
"type": "String",
"description": "Primary shipping address for the customer",
"classification": "PII"
}
]
}
```
- Controls ensure consistent governance and security, e.g., encryption of PII.
- Metadata annotations provide governance information (e.g., data steward, compliance requirements).
- CalmHub Visualization visually represents nodes, interfaces, data flows, lineage, and metadata.
- Patterns facilitate reusable, standardized documentation and controls, such as a “PII Secure Transfer” pattern.
## User Stories:
- As an architect, I want to clearly document the fields and structure of data exchanged between components to ensure clarity and alignment across teams.
- As a compliance officer, I want to identify and track sensitive data (e.g., PII) throughout the system to enforce data governance controls effectively.
- As a developer, I need detailed information about data fields and their movement to debug and optimize components efficiently.
- As a platform engineer, I want to visualize data lineage and flows easily to manage system-wide impacts and changes.
## Current Limitations:
- No built-in capability for detailed field-level data documentation.
- Limited ability to define and track different data movement patterns.
- Lack of lineage tracking capabilities within architecture definitions.
- Insufficient support for annotating data with metadata and governance controls.
## Proposed Implementation:
1. Schema Extensions: Add field-level data element definitions to the CALM schema, supporting detailed metadata (name, type, description, classification).
2. Data Movement Modes: Introduce explicit specification of data movement modes (real-time, streaming, batch) within relationships.
3. Data Lineage and Controls: Enable tracking of data elements across interactions, including transformations and associated governance or security controls.
4. CalmHub Visualization: Enhance CalmHub to visualize detailed data flows, including lineage, metadata, and controls.
## Testing Strategy:
- Unit tests validating schema definitions, data movement modes, and metadata.
- Integration tests verifying proper lineage and control documentation.
- End-to-end tests ensuring accurate visualization and usability in CalmHub.
## Documentation Requirements:
- User Guide updates on creating and interpreting data documentation and movement definitions.
- Developer documentation detailing schema extensions and metadata management.
- Architecture documentation demonstrating practical examples of field-level data documentation and movement tracking.
## Implementation Checklist:
- Extend CALM schema for detailed data documentation.
- Implement explicit data movement mode specifications.
- Enable data lineage tracking and governance metadata annotations.
- Integrate and test visualization capabilities in CalmHub.
- Update user, developer, and architectural documentation.
Implementing this feature will significantly improve how teams document, understand, and manage data through architectural decisions, enhancing compliance, operational clarity, and overall system reliability.
Contributor guide
Assessment
This issue has not been assessed yet.