apache / apache/gravitino

Machine Learning Feature Store Support for Apache Gravitino

Open
#10,594 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
339

Description

### Describe the feature

Attached document proposes adding **ML Feature Store** support to Gravitino—enabling Gravitino to serve as a unified metadata management layer for ML features, bridging the gap between data engineering and machine learning workflows. Feature store metadata is the natural complement to model metadata: models consume features, and features are derived from underlying data assets that Gravitino already manages.

[ML feature store proposal document](https://docs.google.com/document/d/11NYd3KWxryCFxuG887kJ3d0p87leNyqK/edit?usp=sharing&ouid=114123410449326225963&rtpof=true&sd=true)

The doc contains details on:
**Motivation** — Why Gravitino is uniquely positioned (federated metadata, existing model catalog, governance infra)
**Metadata Model** — `FeatureEntity`, `FeatureGroup`, `Feature`, `FeatureGroupVersion`, `FeatureView` with full field definitions
**API Design** — `FeatureStoreCatalog` Java interface with CRUD for all entities, following Gravitino's dispatcher/change-object patterns
**REST API** — Full endpoint specification with example request/response JSON
**Java & Python SDKs** — Code examples showing end-to-end workflows
**Storage** — Database table schemas for MySQL/PostgreSQL/H2 persistence
**Federated Connectors** — Architecture for Feast, Tecton, Hopsworks, SageMaker connectors (Feast as priority)
**Model Catalog Integration** — Feature-model lineage and impact analysis
**Engine Connectors** — Spark, Trino, and Python framework integration
**Governance** — RBAC privileges, tag-based governance, audit events
**CLI & Web UI** — Command examples and UI wireframe descriptions
**Implementation Plan** — 6-phase rollout with task breakdowns
**Testing Strategy** — Unit and integration test plan following Gravitino conventions

### Motivation

Machine learning teams face a critical challenge: managing the lifecycle of **features**—the processed, structured data inputs that ML models consume. In production environments, feature engineering represents 60-80% of the effort in building ML systems, yet features remain one of the least managed assets in the data ecosystem.

### Describe the solution

Add a new FEATURE catalog type to Gravitino that manages feature metadata (not data) — acting as a unified registry and governance layer.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the linked ML feature store proposal, then compare its FeatureStoreCatalog interface and dispatcher/change-object patterns with Gravitino's existing model catalog. Review the proposed REST, SDK, storage, connector, governance, and testing sections; the work is done only when the agreed rollout scope and implementation plan are completed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql, postgresql, python, spark
Domain
backend-api-design, data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.