Three staging/compatibility-view models fail on BigQuery due to hardcoded assumptions (v0.8.1)
- Dominant language
- No language data
- Stars
- 10
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
**Environment:**
- dbt Core 1.9.0, `dbt-bigquery` adapter 1.9.1
- `fivetran/sap` 0.8.1 (bugs 1 and 2 also reproduce on 0.7.0)
---
**Bug 1 — `stg_sap__coep` and `stg_sap__prps` hard-reference `_fivetran_sap_archived` with no fallback**
Both models skip `fivetran_utils.fill_staging_columns` and instead do `select *` from their `_tmp` model, then reference every expected column directly, including `_fivetran_sap_archived`:
```sql
fields as (
select
_fivetran_synced,
_fivetran_sap_archived, -- fails if this column doesn't exist upstream
...
```
That column only exists when Fivetran's "SAP Archiving job users" feature is enabled for the connector. Without it, the reference fails at compile time:
```
Unrecognized name: _fivetran_sap_archived
```
`stg_sap__prps` fails the same way on four more columns (`settlementobject`, `settlmtruleinherited`, `prps_status`, `settlementelement`) listed in `get_prps_columns()` but not present on the real table.
Other staging models (`stg_sap__cosp_bak`, `stg_sap__finsc_cmp_versnd`, `stg_sap__t006`, etc.) already use `fill_staging_columns`, which NULL-fills any missing canonical column instead of erroring.
**Suggested fix:** rewrite both models to use `fivetran_utils.fill_staging_columns` like their siblings.
---
**Bug 2 — `bsad` (and `bsid`) cast BYTES columns straight to STRING**
`bseg.hist_tax_factor`/`hist_tax_factor1`/`hist_tax_factor2`/`hist_tax_factor3` (and the same four columns on `bsad_bck`) are replicated as native `BYTES`, not `STRING`. `compatibility_views/mart/bsad.sql` does:
```sql
cast(i.hist_tax_factor1 as {{ dbt.type_string() }}) as hist_tax_factor1,
```
on all four columns, in both CTEs. `CAST(BYTES AS STRING)` requires valid UTF-8 in BigQuery and throws otherwise:
```
Invalid cast of bytes to UTF8 string
```
Confirmed 100% of rows in `bseg` (171,221,722 / 171,221,722) fail this cast on all four columns. Not BigQuery-specific — Snowflake's `CAST(BINARY AS VARCHAR)` has the same UTF-8 validation; only Spark/Databricks tolerates it silently.
**Suggested fix:** `SAFE_CAST` instead of `CAST` on these four columns.
---
**Bug 3 — `coss` mixes a hardcoded parameterized type with the package's own templated type**
`compatibility_views/mart/coss.sql` uses `{{ dbt.type_numeric() }}` everywhere except one `CASE`:
```sql
case
when v_coss_wdv_5.tsgjahr0 = 0 then cast(0 as {{ dbt.type_numeric() }})
else cast((v_coss_wdv_5.tsgjahr0 - 1990) * 365 * 24 * 60 * 60 as decimal(16, 0))
end as tsgjahr1,
```
The two branches resolve to different types — unparameterized `NUMERIC` vs. explicit `DECIMAL(16, 0)`. `tsgjahr1` is later nested inside another `CAST` downstream, which BigQuery rejects:
```
Parameterized types are not allowed in CAST expressions
```
Not data-dependent — fails on any row count, including zero.
**Suggested fix:** change `decimal(16, 0)` to `{{ dbt.type_numeric() }}`.
---
**Workaround used in the meantime:** project-level overrides (`+enabled: false` on the package model, replaced with a locally-maintained copy with the fix) for all four affected models.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read the staging models stg_sap__coep and stg_sap__prps alongside sibling models using fill_staging_columns, then inspect compatibility_views/mart/bsad.sql and coss.sql. Verify the three reported fixes against BigQuery: missing columns are safely filled, the four BYTES fields no longer fail on invalid UTF-8, and tsgjahr1 uses the package's templated numeric type; done means all four models compile without the reported errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, sql
- Domain
- data, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100