apache / apache/ossie

Add a converter for Alibaba Cloud Hologres Semantic View

Open
#319 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.1k
Forks
267
Avg merge
4d 20h
Merged PRs (30d)
24

Description

## Summary

I would like to contribute a bidirectional converter between Apache Ossie and **Alibaba Cloud Hologres Semantic View**, a feature available since Hologres V5.0.0.

Per CONTRIBUTING.md I am opening this issue first to discuss the approach before submitting a PR.

## Background

A Hologres Semantic View is an in-database object declaring physical tables, their relationships, business dimensions and metrics. Queries reference dimensions and `AGG(metric)` instead of repeating joins and aggregations, and the engine aggregates each metric within its own minimal join subtree so a one-to-many join cannot inflate a total. Hologres also publishes a structured YAML model for every view, intended for BI tools and NL2SQL agents.

## Why it maps cleanly onto Ossie

| Ossie | Hologres Semantic View |
|---|---|
| `dataset` + `source` + `primary_key` | `TABLES (alias AS schema.table PRIMARY KEY (...))` |
| `relationship` (`from` many -> `to` one) | `RELATIONSHIPS (name AS from(cols) REFERENCES to(cols))` |
| `dataset.fields[]` | `DIMENSIONS (alias.name AS expr)` |
| model-level `metrics[]` | `METRICS (owner.name AS agg_expr)` |

The relationship model is a particularly close fit: Hologres requires `from` to be the many side and the `REFERENCES` target to be the referenced table's primary key, which matches Ossie's existing FK-pair semantics exactly.

## Proposed shape

The two directions are asymmetric, because of how Hologres exposes definitions:

- **Export (Ossie -> Hologres)** emits `CREATE SEMANTIC VIEW` **SQL DDL text**. Hologres has no YAML import function, so DDL is the only way to create a view. This would be the first converter in the repo whose output is not YAML — I would like feedback on whether that is acceptable, or whether it should be shaped differently.
- **Import (Hologres -> Ossie)** consumes the `model_yaml` that Hologres publishes in the `hologres.hg_semantic_view_properties` system table.

## No specification change needed

**Edited:** an earlier version of this issue proposed adding a `HOLOGRES` token to the `Dialect` enum, motivated by wanting to label PostgreSQL syntax such as `col::text` honestly rather than as `ANSI_SQL`. On review that motivation does not hold up, so **the proposal now touches no spec files at all** — only `converters/hologres/`, its CI workflow, and one row each in `converters/README.md` and `ROADMAP.md`.

Three reasons, in case they are useful to others considering a vendor dialect token:

1. **The portable spelling is always available.** Hologres accepts `CAST(x AS TEXT)`, and sqlglot normalizes `x::text` to it in both directions, so the shorthand never reaches the output.
2. **A per-expression portability test is unsound.** I tried deciding the label by comparing sqlglot's postgres rendering against its default rendering. sqlglot's default dialect is not ANSI SQL, so this passed `ILIKE` as portable while flagging the standard `SUBSTRING`, `EXTRACT` and `DATE_TRUNC` as vendor-specific.
3. **Over-labelling costs more than it saves.** A converter that looks for an `ANSI_SQL` expression and finds none *drops the field* (`converters/databricks/src/ossie_databricks/ossie_to_metric_view.py`). Tagging a `DATE_TRUNC` dimension as vendor-specific would lose it silently, whereas an optimistic `ANSI_SQL` label at worst surfaces as a SQL error on the target engine.

So everything is read and written as `ANSI_SQL`, and the residual PostgreSQL-only syntax (`j -> 'k'`, `s ~ 'pattern'`, the 1-based `arr[1]`) is labelled slightly optimistically, which the converter README states plainly. This follows the same reasoning as the NVIDIA GSF converter: anything else stays `ANSI_SQL` rather than being labelled inaccurately.

The `HOLOGRES` **vendor** name used for the `custom_extensions` stash is unaffected, since the Ossie `Vendor` field is free-form.

## Known limitations to document

Hologres constrains what a Semantic View can express, so some valid Ossie models cannot be converted. Notably, using the repository's own `examples/tpcds_semantic_model.yaml`, 2 of its 5 metrics (`customer_lifetime_value`, `store_productivity`) are cross-dataset ratios and have no Semantic View form:

- definition expressions must be row-level over a **single** table
- aggregates limited to `count` / `sum` / `avg` / `min` / `max`
- no derived, ratio, or filtered metrics
- a `REFERENCES` target must be the target table's primary key
- no `CREATE OR REPLACE` / `ALTER SEMANTIC VIEW`

My preference is to fail closed and name the offending field, with an opt-in flag to skip unconvertible metrics.

One limitation is worth flagging because it is not in the Hologres documentation: a **bare top-level operator in a definition is a syntax error**. `a || b`, `a + 1` and `a::text` are all rejected in a `DIMENSIONS` clause, while `(a || b)`, `(a + 1)` and `cast(a as text)` are accepted — even though the same operators are fine inside a function call's argument list. The converter adds the required parentheses and strips them back off on import.

## Status

I have a working implementation, verified against a real Hologres 5.0.0 instance: the generated DDL executes, the resulting views return correct results (including the fan-out case where a joined detail table must not inflate a total), and importing Hologres' own readback of that DDL reproduces the original Ossie model. Happy to open the PR once there is agreement on the approach — the main open question is now the DDL-rather-than-YAML output.

Disclosure: this implementation was produced with AI assistance. In line with the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html), I have reviewed the code and take full responsibility for it; the behaviour described above was verified by me against a live instance.

Contributor guide

Open the contributing guide

Research direction

Read CONTRIBUTING.md and inspect the existing patterns under converters/, especially converters/hologres/ if available, along with converters/README.md and ROADMAP.md. Compare the proposed import/export behavior with examples/tpcds_semantic_model.yaml and the documented Hologres limitations. Done means agreement on the DDL output approach, a working bidirectional converter, and documented handling of unsupported models.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.