apache / apache/ossie

dbt osi-to-msi silently corrupts composite primary and unique keys

Closed
#311 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.1k
Forks
267
Avg merge
4d 20h
Merged PRs (30d)
24

Description

## Problem

The dbt `OSIToMSIConverter` flattens every Ossie key group into a set of columns. For a composite primary key such as `[tenant_id, order_id]`, it emits two independent MetricFlow `PRIMARY` entities. The reverse conversion then keeps only the final primary entity, silently changing the key to `[order_id]`. Composite unique keys are likewise converted into independent singleton unique keys.

## Reproduction

On `main` at `88e0011148283302c9a04cd0287e00e0b9d87354`, convert a valid dataset containing:

```yaml
primary_key: [tenant_id, order_id]
```

Observed after Ossie -> MSI:

```text
entities=[(tenant_id, primary), (order_id, primary)]
```

Observed after Ossie -> MSI -> Ossie:

```yaml
primary_key: [order_id]
```

A singleton primary key round-trips unchanged.

## Root cause

`converters/dbt/src/ossie_dbt/osi_to_msi.py::_build_key_sets` converts `primary_key` to a set and flattens all `unique_keys` groups. `_classify_field` then treats every member as independently primary or unique. `MSIToOSIConverter._extract_keys` has no grouping information to reconstruct.

## Impact

Composite keys are common for tenant-scoped and line-item datasets. Emitting each component as independently unique can produce incorrect join cardinality or aggregations, while round trips silently lose key components.

## Proposed fix

Until MSI can represent and round-trip grouped keys losslessly, fail Ossie -> MSI conversion clearly when a dataset contains a composite primary or unique key. Keep singleton key behavior unchanged and add regression tests for both composite key types. This prevents generating a plausible but semantically incorrect manifest.

Contributor guide

Open the contributing guide

Research direction

Start in converters/dbt/src/ossie_dbt/osi_to_msi.py, reading _build_key_sets and _classify_field, then inspect MSIToOSIConverter._extract_keys to understand the missing grouping information. Add regression coverage for composite primary and unique keys while preserving singleton behavior. Done means composite keys fail clearly during Ossie-to-MSI conversion instead of producing misleading independent entities.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.