[EPIC] Introducing metadata-driven action system in Gravitino
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Describe the proposal
Metadata is the brain of the data, also it is the treasure for the company. Currently, Gravitino already supports managing structured data (table), unstructured data (fileset), messaging data (topic), and ML models (model), so what is the next step, how to leverage metadata to empower the business?
Intuitively, the next step is to make decisions and take actions based on the managed metadata in Gravitino. With this, we can support lots of different scenarios, for example, like:
1. Table, column, fileset (sub-directories), and model data clean based on the TTL policy.
2. Multiple location fileset synchronization based on the local-remote R/W strategy.
3. Table maintenance, like data file compaction, data file rewrite, snapshot expiration, orphan file removal, etc.
4. Regulation supports, like column masking/pruning, column deletion, data encryption.
5. Various actions that are based on the existing metadata.
So, Gravitino is already the brain of the data, the next step is to build arms and legs for Gravitino to truly empower the business. With this, we will kick off a new key project in Gravitino - a metadata-driven action system.
## Goal
1. Build a generic action system in Gravitino. This action system is not only limited to table maintenance but also can support other various actions based on the provided metadata in Gravitino. Also, users can build and submit their actions with our provided framework.
2. Build the framework in a self-contained, generic, and atomic way:
* Each component in this framework is self-contained, and can be reused in other scenarios.
* Each component should provide REST APIs, and can be integrated with an agentic framework, like MCP, to make it a tool or MCP server.
Contributor guide
Assessment
This issue has not been assessed yet.