Add a converter for Alibaba Cloud Hologres Semantic View
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 267
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 24
Description
## Summary
I would like to contribute a bidirectional converter between Apache Ossie and **Alibaba Cloud Hologres Semantic View**, a feature available since Hologres V5.0.0.
Per CONTRIBUTING.md I am opening this issue first to discuss the approach before submitting a PR.
## Background
A Hologres Semantic View is an in-database object declaring physical tables, their relationships, business dimensions and metrics. Queries reference dimensions and `AGG(metric)` instead of repeating joins and aggregations, and the engine aggregates each metric within its own minimal join subtree so a one-to-many join cannot inflate a total. Hologres also publishes a structured YAML model for every view, intended for BI tools and NL2SQL agents.
## Why it maps cleanly onto Ossie
| Ossie | Hologres Semantic View |
|---|---|
| `dataset` + `source` + `primary_key` | `TABLES (alias AS schema.table PRIMARY KEY (...))` |
| `relationship` (`from` many -> `to` one) | `RELATIONSHIPS (name AS from(cols) REFERENCES to(cols))` |
| `dataset.fields[]` | `DIMENSIONS (alias.name AS expr)` |
| model-level `metrics[]` | `METRICS (owner.name AS agg_expr)` |
The relationship model is a particularly close fit: Hologres requires `from` to be the many side and the `REFERENCES` target to be the referenced table's primary key, which matches Ossie's existing FK-pair semantics exactly.
## Proposed shape
The two directions are asymmetric, because of how Hologres exposes definitions:
- **Export (Ossie -> Hologres)** emits `CREATE SEMANTIC VIEW` **SQL DDL text**. Hologres has no YAML import function, so DDL is the only way to create a view. This would be the first converter in the repo whose output is not YAML — I would like feedback on whether that is acceptable, or whether it should be shaped differently.
- **Import (Hologres -> Ossie)** consumes the `model_yaml` that Hologres publishes in the `hologres.hg_semantic_view_properties` system table.
## No specification change needed
**Edited:** an earlier version of this issue proposed adding a `HOLOGRES` token to the `Dialect` enum, motivated by wanting to label PostgreSQL syntax such as `col::text` honestly rather than as `ANSI_SQL`. On review that motivation does not hold up, so **the proposal now touches no spec files at all** — only `converters/hologres/`, its CI workflow, and one row each in `converters/README.md` and `ROADMAP.md`.
Three reasons, in case they are useful to others considering a vendor dialect token:
1. **The portable spelling is always available.** Hologres accepts `CAST(x AS TEXT)`, and sqlglot normalizes `x::text` to it in both directions, so the shorthand never reaches the output.
2. **A per-expression portability test is unsound.** I tried deciding the label by comparing sqlglot's postgres rendering against its default rendering. sqlglot's default dialect is not ANSI SQL, so this passed `ILIKE` as portable while flagging the standard `SUBSTRING`, `EXTRACT` and `DATE_TRUNC` as vendor-specific.
3. **Over-labelling costs more than it saves.** A converter that looks for an `ANSI_SQL` expression and finds none *drops the field* (`converters/databricks/src/ossie_databricks/ossie_to_metric_view.py`). Tagging a `DATE_TRUNC` dimension as vendor-specific would lose it silently, whereas an optimistic `ANSI_SQL` label at worst surfaces as a SQL error on the target engine.
So everything is read and written as `ANSI_SQL`, and the residual PostgreSQL-only syntax (`j -> 'k'`, `s ~ 'pattern'`, the 1-based `arr[1]`) is labelled slightly optimistically, which the converter README states plainly. This follows the same reasoning as the NVIDIA GSF converter: anything else stays `ANSI_SQL` rather than being labelled inaccurately.
The `HOLOGRES` **vendor** name used for the `custom_extensions` stash is unaffected, since the Ossie `Vendor` field is free-form.
## Known limitations to document
Hologres constrains what a Semantic View can express, so some valid Ossie models cannot be converted. Notably, using the repository's own `examples/tpcds_semantic_model.yaml`, 2 of its 5 metrics (`customer_lifetime_value`, `store_productivity`) are cross-dataset ratios and have no Semantic View form:
- definition expressions must be row-level over a **single** table
- aggregates limited to `count` / `sum` / `avg` / `min` / `max`
- no derived, ratio, or filtered metrics
- a `REFERENCES` target must be the target table's primary key
- no `CREATE OR REPLACE` / `ALTER SEMANTIC VIEW`
My preference is to fail closed and name the offending field, with an opt-in flag to skip unconvertible metrics.
One limitation is worth flagging because it is not in the Hologres documentation: a **bare top-level operator in a definition is a syntax error**. `a || b`, `a + 1` and `a::text` are all rejected in a `DIMENSIONS` clause, while `(a || b)`, `(a + 1)` and `cast(a as text)` are accepted — even though the same operators are fine inside a function call's argument list. The converter adds the required parentheses and strips them back off on import.
## Status
I have a working implementation, verified against a real Hologres 5.0.0 instance: the generated DDL executes, the resulting views return correct results (including the fan-out case where a joined detail table must not inflate a total), and importing Hologres' own readback of that DDL reproduces the original Ossie model. Happy to open the PR once there is agreement on the approach — the main open question is now the DDL-rather-than-YAML output.
Disclosure: this implementation was produced with AI assistance. In line with the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html), I have reviewed the code and take full responsibility for it; the behaviour described above was verified by me against a live instance.
Contributor guide
Research direction
Read CONTRIBUTING.md and inspect the existing patterns under converters/, especially converters/hologres/ if available, along with converters/README.md and ROADMAP.md. Compare the proposed import/export behavior with examples/tpcds_semantic_model.yaml and the documented Hologres limitations. Done means agreement on the DDL output approach, a working bidirectional converter, and documented handling of unsupported models.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- data, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100