msi_to_osi silently discards config.meta instead of mapping it to custom_extensions
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 267
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 24
Description
`MSIToOSIConverter` silently discards `config.meta` from metrics, dimensions and semantic models. MSI carries the metadata, Ossie has a place to put it (`custom_extensions`), but the converter never reads one or writes the other — and unlike other lossy paths in this converter, no `ConverterIssue` is emitted.
**Root cause**
`converters/dbt/src/ossie_dbt/msi_to_osi.py`. Each OSI object is built from a fixed handful of attributes:
```python
OSIMetric(
name=metric.name,
expression=self._make_expression(expr),
description=metric.description,
)
```
```python
return OSIField(
name=dim.name,
expression=self._make_expression(expr),
dimension=OSIDimension(is_time=is_time),
label=dim.label,
description=dim.description,
)
```
`grep -n '\.config\|meta' converters/dbt/src/ossie_dbt/msi_to_osi.py` returns nothing — `.config` is never accessed anywhere in the converter.
Both sides of the mapping already exist:
- **Source:** `PydanticMetric`, `PydanticSemanticModel` and `PydanticDimension` all expose a `config` field of type `PydanticSemanticLayerElementConfig`, whose `meta` is an arbitrary dict. dbt writes user-supplied `config.meta` into `semantic_manifest.json`, so it is present in the object the converter is handed.
- **Destination:** `custom_extensions` is defined on `Field`, `Metric`, `Dataset`, `Relationship` and `SemanticModel` in `core-spec/osi-schema.json`, and `Vendor` is documented as "Any string value is accepted", with `DBT` given as an example.
**Repro** (HEAD `88e0011148283302c9a04cd0287e00e0b9d87354`), using the repo's own test helpers:
```python
import sys
sys.path.insert(0, "src"); sys.path.insert(0, "."); sys.path.insert(0, "../../python/src")
from ossie_dbt.msi_to_osi import MSIToOSIConverter
from tests.helpers import _entity, _manifest, _simple_metric
from metricflow_semantic_interfaces.implementations.element_config import PydanticSemanticLayerElementConfig
from metricflow_semantic_interfaces.implementations.elements.dimension import PydanticDimension
from metricflow_semantic_interfaces.implementations.elements.measure import PydanticMeasure
from metricflow_semantic_interfaces.test_utils import semantic_model_with_guaranteed_meta
from metricflow_semantic_interfaces.type_enums import AggregationType, DimensionType, EntityType
meta = {"value_format": '[>=1000000]$#,##0.00,,"M";$#,##0.00', "group_label": "Booking Costs"}
sm = semantic_model_with_guaranteed_meta(
name="bookings",
entities=[_entity("booking", EntityType.PRIMARY, "booking_id")],
measures=[PydanticMeasure(name="amount", agg=AggregationType.SUM, expr="amount_eur")],
dimensions=[PydanticDimension(name="status", type=DimensionType.CATEGORICAL,
config=PydanticSemanticLayerElementConfig(meta=meta))],
)
metric = _simple_metric("total_amount", "amount")
metric.config = PydanticSemanticLayerElementConfig(meta=meta)
res = MSIToOSIConverter().convert(_manifest(semantic_models=[sm], metrics=[metric]))
osi_metric = res.output.semantic_model[0].metrics[0]
osi_field = [f for f in res.output.semantic_model[0].datasets[0].fields if f.name == "status"][0]
print("metric.custom_extensions:", osi_metric.custom_extensions)
print("field.custom_extensions :", osi_field.custom_extensions)
print("ISSUES:", res.issues)
```
Run from `converters/dbt/`.
Observed:
```
metric.custom_extensions: None
field.custom_extensions : None
ISSUES: []
```
Expected: the `meta` dict carried through as a `custom_extensions` entry — or, if that is deliberately out of scope, a `ConverterIssue` so the loss is visible.
**Why this matters**
`config.meta` is the only place dbt users can attach metadata that the MSI spec does not model — units, display formatting, grouping, ownership, links back to a source system. For anyone migrating an existing BI semantic layer into dbt it tends to be where all the presentation metadata ends up, precisely because the spec has nowhere else for it. Discarding it at the Ossie boundary means that content cannot reach any downstream consumer, even ones that would understand it.
It is also inconsistent with how this converter handles its other lossy paths: `CUMULATIVE` metrics emit `CUMULATIVE_SEMANTICS_LOSS`, conversion metrics emit `CONVERSION_METRIC_DROPPED`, natural entities emit `NATURAL_ENTITY_DROPPED`. This loss is silent.
**Suggested fix**
Map `config.meta` onto a single `custom_extensions` entry when non-empty, e.g.
```python
def _meta_extensions(config) -> Optional[List[OSICustomExtension]]:
meta = getattr(config, "meta", None) if config else None
if not meta:
return None
return [OSICustomExtension(vendor_name="DBT", data=json.dumps(meta))]
```
applied in `_convert_metrics`, `_convert_dimension`, `_convert_entity`, `_convert_measure` and `_convert_dataset`. `CustomExtension.data` is a JSON string, so the dict serialises directly.
Happy to send a PR if the approach sounds right. Two things worth deciding first:
1. **Vendor name.** `DBT` matches the schema's own example, but `COMMON` may be more appropriate if the intent is that any consumer may read it rather than dbt-specific tooling.
2. **Round-tripping.** `osi_to_msi` would need the inverse to make this lossless in both directions. Happy to include that in the same PR.
**Note:** the same `msi_to_osi` implementation is vendored in `dbt-labs/metricflow` as `metricflow/converters/msi_to_osi.py` (byte-identical apart from one word in a docstring) and ships inside dbt-core 1.12, where it generates `target/osi_document.json`. The same gap exists there.
Related: #111 covers `custom_extensions` being dropped again on the Snowflake side, so both would need addressing for metadata to survive a full dbt → Ossie → Snowflake conversion.
Contributor guide
Research direction
Start in converters/dbt/src/ossie_dbt/msi_to_osi.py, focusing on _convert_metrics, _convert_dimension, _convert_entity, _convert_measure, and _convert_dataset. Run the supplied reproduction from converters/dbt using the tests.helpers fixtures, then inspect the existing ConverterIssue patterns. Done means config.meta is preserved in custom_extensions or its loss is explicitly reported, with the vendor and reverse-mapping scope resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100