Machine Learning Feature Store Support for Apache Gravitino
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 339
Description
### Describe the feature
Attached document proposes adding **ML Feature Store** support to Gravitino—enabling Gravitino to serve as a unified metadata management layer for ML features, bridging the gap between data engineering and machine learning workflows. Feature store metadata is the natural complement to model metadata: models consume features, and features are derived from underlying data assets that Gravitino already manages.
[ML feature store proposal document](https://docs.google.com/document/d/11NYd3KWxryCFxuG887kJ3d0p87leNyqK/edit?usp=sharing&ouid=114123410449326225963&rtpof=true&sd=true)
The doc contains details on:
**Motivation** — Why Gravitino is uniquely positioned (federated metadata, existing model catalog, governance infra)
**Metadata Model** — `FeatureEntity`, `FeatureGroup`, `Feature`, `FeatureGroupVersion`, `FeatureView` with full field definitions
**API Design** — `FeatureStoreCatalog` Java interface with CRUD for all entities, following Gravitino's dispatcher/change-object patterns
**REST API** — Full endpoint specification with example request/response JSON
**Java & Python SDKs** — Code examples showing end-to-end workflows
**Storage** — Database table schemas for MySQL/PostgreSQL/H2 persistence
**Federated Connectors** — Architecture for Feast, Tecton, Hopsworks, SageMaker connectors (Feast as priority)
**Model Catalog Integration** — Feature-model lineage and impact analysis
**Engine Connectors** — Spark, Trino, and Python framework integration
**Governance** — RBAC privileges, tag-based governance, audit events
**CLI & Web UI** — Command examples and UI wireframe descriptions
**Implementation Plan** — 6-phase rollout with task breakdowns
**Testing Strategy** — Unit and integration test plan following Gravitino conventions
### Motivation
Machine learning teams face a critical challenge: managing the lifecycle of **features**—the processed, structured data inputs that ML models consume. In production environments, feature engineering represents 60-80% of the effort in building ML systems, yet features remain one of the least managed assets in the data ecosystem.
### Describe the solution
Add a new FEATURE catalog type to Gravitino that manages feature metadata (not data) — acting as a unified registry and governance layer.
### Additional context
_No response_
Contributor guide
Research direction
Start with the linked ML feature store proposal, then compare its FeatureStoreCatalog interface and dispatcher/change-object patterns with Gravitino's existing model catalog. Review the proposed REST, SDK, storage, connector, governance, and testing sections; the work is done only when the agreed rollout scope and implementation plan are completed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql, postgresql, python, spark
- Domain
- backend-api-design, data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100