fivetran / fivetran/dbt_sap

Three staging/compatibility-view models fail on BigQuery due to hardcoded assumptions (v0.8.1)

Open
#53 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
10
Forks
5
PR merge metrics
No merged PRs in 30d

Description

**Environment:**
- dbt Core 1.9.0, `dbt-bigquery` adapter 1.9.1
- `fivetran/sap` 0.8.1 (bugs 1 and 2 also reproduce on 0.7.0)

---

**Bug 1 — `stg_sap__coep` and `stg_sap__prps` hard-reference `_fivetran_sap_archived` with no fallback**

Both models skip `fivetran_utils.fill_staging_columns` and instead do `select *` from their `_tmp` model, then reference every expected column directly, including `_fivetran_sap_archived`:

```sql
fields as (
select
_fivetran_synced,
_fivetran_sap_archived, -- fails if this column doesn't exist upstream
...
```

That column only exists when Fivetran's "SAP Archiving job users" feature is enabled for the connector. Without it, the reference fails at compile time:
```
Unrecognized name: _fivetran_sap_archived
```
`stg_sap__prps` fails the same way on four more columns (`settlementobject`, `settlmtruleinherited`, `prps_status`, `settlementelement`) listed in `get_prps_columns()` but not present on the real table.

Other staging models (`stg_sap__cosp_bak`, `stg_sap__finsc_cmp_versnd`, `stg_sap__t006`, etc.) already use `fill_staging_columns`, which NULL-fills any missing canonical column instead of erroring.

**Suggested fix:** rewrite both models to use `fivetran_utils.fill_staging_columns` like their siblings.

---

**Bug 2 — `bsad` (and `bsid`) cast BYTES columns straight to STRING**

`bseg.hist_tax_factor`/`hist_tax_factor1`/`hist_tax_factor2`/`hist_tax_factor3` (and the same four columns on `bsad_bck`) are replicated as native `BYTES`, not `STRING`. `compatibility_views/mart/bsad.sql` does:
```sql
cast(i.hist_tax_factor1 as {{ dbt.type_string() }}) as hist_tax_factor1,
```
on all four columns, in both CTEs. `CAST(BYTES AS STRING)` requires valid UTF-8 in BigQuery and throws otherwise:
```
Invalid cast of bytes to UTF8 string
```
Confirmed 100% of rows in `bseg` (171,221,722 / 171,221,722) fail this cast on all four columns. Not BigQuery-specific — Snowflake's `CAST(BINARY AS VARCHAR)` has the same UTF-8 validation; only Spark/Databricks tolerates it silently.

**Suggested fix:** `SAFE_CAST` instead of `CAST` on these four columns.

---

**Bug 3 — `coss` mixes a hardcoded parameterized type with the package's own templated type**

`compatibility_views/mart/coss.sql` uses `{{ dbt.type_numeric() }}` everywhere except one `CASE`:
```sql
case
when v_coss_wdv_5.tsgjahr0 = 0 then cast(0 as {{ dbt.type_numeric() }})
else cast((v_coss_wdv_5.tsgjahr0 - 1990) * 365 * 24 * 60 * 60 as decimal(16, 0))
end as tsgjahr1,
```
The two branches resolve to different types — unparameterized `NUMERIC` vs. explicit `DECIMAL(16, 0)`. `tsgjahr1` is later nested inside another `CAST` downstream, which BigQuery rejects:
```
Parameterized types are not allowed in CAST expressions
```
Not data-dependent — fails on any row count, including zero.

**Suggested fix:** change `decimal(16, 0)` to `{{ dbt.type_numeric() }}`.

---

**Workaround used in the meantime:** project-level overrides (`+enabled: false` on the package model, replaced with a locally-maintained copy with the fix) for all four affected models.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read the staging models stg_sap__coep and stg_sap__prps alongside sibling models using fill_staging_columns, then inspect compatibility_views/mart/bsad.sql and coss.sql. Verify the three reported fixes against BigQuery: missing columns are safely filled, the four BYTES fields no longer fail on invalid UTF-8, and tsgjahr1 uses the package's templated numeric type; done means all four models compile without the reported errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, sql
Domain
data, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.