apache / apache/ossie

Add an Apache Superset dataset YAML converter

Open
#268 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.1k
Forks
267
Avg merge
4d 20h
Merged PRs (30d)
24

Description

Hi all,

I would like to propose a bidirectional converter between Apache Ossie semantic models and Apache Superset dataset definitions.

## Motivation

Superset provides a lightweight semantic layer around physical and virtual datasets, calculated columns, and reusable metrics. It also provides supported YAML import/export workflows and REST endpoints for dataset bundles. A converter would let teams move these definitions into Ossie for interchange with other semantic tools and publish an Ossie model back into a Superset-compatible dataset bundle.

Official references:

- https://superset.apache.org/docs/
- https://superset.apache.org/docs/configuration/importing-exporting-datasources/
- https://superset.apache.org/developer-docs/api/datasets/

I searched the open and closed Ossie issues and pull requests and did not find an existing Superset converter proposal.

## Proposed v1 boundary

- Convert static Superset dataset YAML, without requiring a running Superset instance.
- Support physical and SQL-based virtual datasets.
- Map dataset names, descriptions, schemas, tables, columns, calculated-column expressions, temporal flags, and dataset metrics.
- Accept and emit the documented dataset import/export bundle structure.
- Do not include charts, dashboards, saved queries, users, roles, or database credentials.
- Do not execute Jinja templates. Preserve template-bearing SQL or calculated expressions as Superset-specific metadata and report that they cannot be statically interpreted.
- Preserve supported Superset-only properties in a versioned `SUPERSET` custom extension so semantic round-trips do not silently discard them.
- Validate generated Ossie documents against `core-spec/osi-schema.json`.

## Initial mapping

| Superset | Ossie |
|---|---|
| physical or virtual dataset | `dataset` |
| table/schema or virtual SQL | `dataset.source` and Superset extension metadata |
| dataset column | `field` |
| calculated column SQL | `field.expression` |
| temporal column | `field.dimension.is_time` |
| dataset metric | model-level `metric` with owning-dataset metadata |
| dataset description and column descriptions | corresponding Ossie descriptions |

Superset metrics are dataset-scoped, while Ossie metrics are model-scoped. On import, duplicate metric names from different datasets therefore need deterministic qualification while retaining the original name and owner. On export, the converter should use preserved owner metadata and only infer ownership from expressions when exactly one dataset is referenced.

Superset's documented dataset format does not provide an Ossie-style inter-dataset relationship graph. For an Ossie-to-Superset-to-Ossie round-trip, relationships would need to be preserved in the Superset extension or reported as unsupported rather than assigned or discarded silently.

## Proposed project shape

- Distribution: `apache-ossie-superset`
- Python module: `ossie_superset`
- Directory: `converters/superset/`
- Static YAML parsing with an Apache-compatible YAML dependency already acceptable to the project
- Unit fixtures covering physical and virtual datasets, calculated columns, temporal fields, metric-name collisions, Jinja-bearing expressions, and metadata restoration
- Semantic round-trip tests in both directions
- Optional integration validation against a local Superset instance, separate from the static runtime converter

## Questions for maintainers

1. Would this converter and the static dataset-YAML v1 boundary be welcome?
2. Should v1 target Superset's established dataset import/export format only, or account for the newer experimental `SEMANTIC_LAYERS` feature separately?
3. Is preserving Ossie relationships in a `SUPERSET` extension acceptable when Superset cannot represent them natively?
4. Does the proposed Python package naming match the direction for newer converters?

If the direction is welcome, I would like to contribute the converter and first prepare a precise mapping/loss matrix and representative exported fixtures before opening a focused implementation PR for review.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the proposed converters/superset/ directory and ossie_superset package conventions, then inspect core-spec/osi-schema.json and existing converter tests. Prepare the mapping/loss matrix and representative physical and virtual Superset YAML fixtures before implementation. Done means the v1 boundary, extension handling, validation, and semantic round-trip tests are accepted by maintainers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.