[EPIC] Add Policy Management Support to Gravitino
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Describe the proposal
Gravitino manages metadata for various data assets - tables, filesets, ML models, and messaging topics. It provides a unified API that works consistently across different data systems. As data volumes grow in organizations, managing governance becomes increasingly challenging, creating the need for effective policy controls.
To address these challenges, we're adding a policy framework to Gravitino. This framework will let users define rules that govern how metadata and the associated data are managed. Here are several common application scenarios for policy:
- **Data Lifecycle Management** - Specify Time-to-Live (TTL) for Filesets, Tables, and Models
- **Table Maintenance Strategy** - Orchestrating data organization strategies, including compaction, expiring snapshots, and removing orphan files
- **Access Control Enhancement** - Extending basic permissions with conditional, and temporal access constraints
- **Data Compliance** - Establishing validation frameworks with configurable quality thresholds and remediation actions
### Task list
see below
Contributor guide
Assessment
This issue has not been assessed yet.