apache / apache/ossie

Add bidirectional Apache Gravitino converter

Open
#256 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.1k
Forks
267
Avg merge
4d 20h
Merged PRs (30d)
24

Description

Hi all 👋

I'd like to gauge interest in adding a **bidirectional converter** between Apache Ossie semantic models and [Apache Gravitino](https://gravitino.apache.org/) under `converters/gravitino/`.

## Background

[Apache Gravitino](https://gravitino.apache.org/) is a high-performance, geo-distributed, federated metadata lake (Apache Incubating). It provides a unified REST API to manage metadata across many data sources — Hive, JDBC (MySQL/PostgreSQL/Doris/StarRocks), Iceberg, Paimon, Hudi, Kafka, and file-based sources — through a single "catalog of catalogs" abstraction.

Gravitino already knows about **physical metadata** (catalogs, schemas, tables, columns, types, partition keys, tags, properties) for the entire big-data ecosystem. What it does **not** carry is Ossie's **semantic layer** — metrics, cross-dataset relationships, time-dimension markers, and AI context. A converter between the two would let any Ossie-compatible BI/AI tool access the full inventory of a big-data platform through Gravitino's unified API, and conversely let a semantic model authored in Ossie be materialized as table metadata in a big-data catalog.

This is also a **two-Apache-podlings integration** — both Gravitino and Ossie are Apache Incubating projects, and a converter between them benefits both ecosystems.

**I'd like to contribute this converter myself.** I'm a big-data platform engineer with deep experience in the Hive/Spark/Iceberg ecosystem, and I've already studied the existing `converters/polaris/` codebase and prototyped the Gravitino REST API integration. I'm happy to follow the "Contributing a New Converter" steps (vendor entry, both directions, tests, README) and would open a PR once the approach is agreed.

## What's included

- **`converters/gravitino/`** — a Java/Maven module following the pattern of `converters/polaris/`.
- `GravitinoClient` — a thin REST client calling Gravitino's existing API (Java 11 `HttpClient` + Jackson), shared by both directions:
- `GET /api/metalakes/{m}/catalogs` → list catalogs
- `GET /api/metalakes/{m}/catalogs/{c}/schemas` → list schemas
- `GET /api/metalakes/{m}/catalogs/{c}/schemas/{s}/tables` → list tables
- `GET /api/metalakes/{m}/catalogs/{c}/schemas/{s}/tables/{t}` → load table
- `POST /api/metalakes/{m}/catalogs/{c}/schemas/{s}/tables` → create table

### Import (Gravitino → Ossie)

Reads table/schema metadata from a Gravitino metalake and produces an Ossie semantic model YAML.

| Gravitino concept | Ossie concept |
|---|---|
| Metalake + Catalog | `SemanticModel` |
| Schema | dataset grouping within the model |
| Table | `Dataset` |
| `table.name` | `dataset.name` |
| `catalog.schema.table` | `dataset.source` |
| `table.columns[]` | `dataset.fields[]` |
| `column.name` + `column.type` | `field.name` + `expression` (ANSI_SQL) |
| `column.type` = timestamp/date | `field.dimension.is_time = true` |
| `table.partitionKeys[]` | flagged in `field.description` or `custom_extensions` |
| `table.properties` | `dataset.custom_extensions[vendor_name: GRAVITINO]` |
| `table.tags` | candidate source for `dataset.ai_context` |

CLI: `ossie-gravitino-converter import --url URL --metalake M [--catalog C] [--auth ...] -o output.yaml`

### Export (Ossie → Gravitino)

Takes an Ossie semantic model and creates/updates the corresponding tables in Gravitino via its REST API.

| Ossie concept | Gravitino concept |
|---|---|
| `SemanticModel` | Metalake + Catalog (1:1, or configurable mapping) |
| `Dataset` | `Table` |
| `dataset.name` | `table.name` |
| `dataset.source` | resolved to `catalog.schema.table` |
| `dataset.fields[]` | `table.columns[]` |
| `field.expression` (ANSI_SQL) | `column.name` (extracted from expression) |
| `field.dimension.is_time` | `column.type` set to `timestamp` / `date` |
| `dataset.primary_key` | `table.primaryKey` or `table.constraints` |
| `dataset.unique_keys` | `table.constraints` (unique) |
| `dataset.custom_extensions[vendor_name: GRAVITINO]` | `table.properties` (round-trip) |
| `dataset.description` / `field.description` | `table.comment` / `column.comment` |

CLI: `ossie-gravitino-converter export --url URL --metalake M --catalog C ossie_model.yaml [--auth ...] [--dry-run]`

## Design considerations

- **Round-trip fidelity**: `Gravitino → Ossie → Gravitino` should be lossless. Gravitino's relational model does not have a native "relationship" or "metric" concept, so Ossie relationships and metrics are serialized into `custom_extensions[vendor_name: OSSIE]` on the corresponding table, following the round-trip-fidelity pattern used by the Databricks converter (#224).
- **No changes on the Gravitino side**: the converter is a pure client of Gravitino's existing REST API. Gravitino does not need to know about Ossie.
- **Idempotency**: the exporter checks if a table already exists before creating (GET then POST/PUT), so re-running is safe.
- **Type inference**: Ossie fields are expression-based and don't carry explicit physical types. The exporter uses heuristics (field name patterns + `is_time` flag) to infer Gravitino column types, similar to `PolarisExporter.inferIcebergType()`. Gravitino's `External Type` fallback is used when inference fails.
- **Big-data ecosystem gap**: Ossie currently has converters for BI/SaaS tools (dbt, Snowflake, Tableau, GoodData, Omni, …) but **none for big-data catalog/metadata services**. Gravitino is the natural bridge — one converter unlocks Hive, Iceberg, Paimon, Hudi, JDBC, and more.

## Approach

I'd follow the existing converter pattern (closest analogue: `converters/polaris/`):

1. `GravitinoClient` — REST client authenticating via Gravitino's OAuth or basic auth.
2. `GravitinoImporter` — walks metalake → catalog → schema → table, maps each table to an Ossie `Dataset` with fields derived from columns. Time types (`timestamp`, `date`, `timestamptz`) set `dimension.is_time`.
3. `GravitinoExporter` — reads an Ossie model, infers column types, and calls Gravitino's create-table API.
4. Reuse `OsiYamlGenerator` / `OsiModelParser` (or the shared `ossie-common` module if #235 lands first).
5. Gravitino tags/properties that carry semantic hints (e.g. `description`, `owner`) flow into `ai_context` or `description`; non-semantic properties go into `custom_extensions[vendor_name: GRAVITINO]`.
6. CLI + tests (mocked Gravitino API responses with Hive + JDBC-MySQL fixtures) + README, following CONTRIBUTING.md.

Before I open the PR: **is there anything you'd want me to keep in mind on approach or scope?** And should I wait for #235 (Java shared module) to land, or build against the current `converters/polaris/` structure for now?

I'll go ahead and start the implementation either way — just want to make sure I'm aligned with the project's direction.

Thanks!

Contributor guide

Open the contributing guide

Research direction

Start with converters/polaris/ and CONTRIBUTING.md to compare the proposed Java/Maven module structure, then review the round-trip approach in #224 and the shared-module question in #235. The work is done when the importer, exporter, CLI, mocked API tests, and README cover both directions and the proposed metadata mappings.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.