fivetran / fivetran/dbt_netsuite

[BUG] Using `ephemeral` materialization causes all tables to have all null records

Open
#176 1 comment 0 reactions 0 assignees View on GitHub
type:wontfix
Dominant language
No language data
Stars
57
Forks
42
Avg merge
6d 9h
Merged PRs (30d)
2

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the issue

Due to fivetran/dbt_netsuite_source#32, I'm considering `ephemeral` as the materialization type for netsuite_source, so that I can then create my own staging schema that has all of the tables my connector is bringing in.

The only alternatives I can think of is to try to put them in the same schema, which would cause name conflicts, or to have two staging schemas for the same source, which will be confusing for everyone. Since nobody will directly use the staging models from netsuite_source anyway, I figure it's ok to abstract / hide them away in ephemeral models.

When I try to do this, all fields for all records for all models are null. When I look in my `target` folder, I can see the generated SQL is explicitly using `null` as the value for everything.

Is there a way I can use ephemeral as the materialization for netsuite_source?

### Relevant error log or model output

target/compiled/netsuite_source/models/netsuite2/stg_netsuite2__transaction_lines.sql:
```shell

with __dbt__cte__stg_netsuite2__transaction_lines_tmp as (

select *
from [my_db_name_redacted].netsuite_suiteanalytics.transactionline
),base as (

select *
from __dbt__cte__stg_netsuite2__transaction_lines_tmp
),

fields as (

select

cast(null as boolean) as

_fivetran_deleted

,
cast(null as TIMESTAMP) as

_fivetran_synced

,
cast(null as INT) as

id

,
cast(null as INT) as

transaction

,
cast(null as INT) as

linesequencenumber

,
cast(null as TEXT) as

memo

,
cast(null as INT) as

entity

,
cast(null as INT) as

item

,
cast(null as INT) as

class

,
cast(null as INT) as

location

,
cast(null as INT) as

subsidiary

,
cast(null as INT) as

department

,
cast(null as TEXT) as

isclosed

,
cast(null as TEXT) as

isbillable

,
cast(null as TEXT) as

iscogs

,
cast(null as TEXT) as

cleared

,
cast(null as TEXT) as

commitmentfirm

,
cast(null as TEXT) as

mainline

,
cast(null as TEXT) as

taxline

from base
),

final as (

select
_fivetran_synced,
id as transaction_line_id,
transaction as transaction_id,
linesequencenumber as transaction_line_number,
memo,
entity as entity_id,
item as item_id,
class as class_id,
location as location_id,
subsidiary as subsidiary_id,
department as department_id,
isclosed = 'T' as is_closed,
isbillable = 'T' as is_billable,
iscogs = 'T' as is_cogs,
cleared = 'T' as is_cleared,
commitmentfirm = 'T' as is_commitment_firm,
mainline = 'T' as is_main_line,
taxline = 'T' as is_tax_line

--The below macro adds the fields defined within your transaction_lines_pass_through_columns variable into the staging model

from fields
)

select *
from final
```

### Expected behavior

I would expect the CTEs to use the actual fields rather than nulls.

### dbt Project configurations
```
models:

+persist_docs:
relation: true
columns: true

+copy_grants: true

netsuite_source:
+schema: stg_netsuite_suiteanalytics
+materialized: ephemeral

vars:
netsuite_data_model: netsuite2
netsuite2__multibook_accounting_enabled: false
netsuite2__using_vendor_categories: true
```
### Package versions
```
packages:
# - package: calogica/dbt_date
# version: 0.7.2
# included as dependency in fivetran/zendesk

# - package: dbt-labs/dbt_utils
# version: 1.0.0
# included as dependency in fivetran/zendesk

# - package: dbt-labs/dbt_project_evaluator
# version: 0.3.0

# - package: calogica/dbt_expectations
# version: 0.8.2

- package: dbt-labs/codegen
version: 0.9.0

- package: fivetran/zendesk
version: [">=0.10.0", "<0.11.0"]
# includes dependencies to calogica/dbt_date and dbt-labs/dbt_utils

- package: fivetran/netsuite
version: [">=0.7.0", "<0.8.0"]
# includes dependencies to fivetran/netsuite_source, fivetran/fivetran_utils, dbt-labs/dbt_utils, and dbt-labs/spark_utils
```
### What database are you using dbt with?

snowflake 7.12.3

### dbt Version
```
Core:
- installed: 1.4.5
- latest: 1.4.5 - Up to date!

Plugins:
- snowflake: 1.4.2 - Up to date!
```
### Additional Context

If I switch `materialized` to `view` then it all works great.

I'm using Python 3.10.10.

### Are you willing to open a PR to help address this issue?

- [ ] Yes.
- [ ] Yes, but I will need assistance and will schedule time during our [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance
- [ ] No.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the generated SQL at target/compiled/netsuite_source/models/netsuite2/stg_netsuite2__transaction_lines.sql and compare it with the netsuite_source models under the ephemeral configuration in dbt_project.yml. Trace why the fields CTE emits null values while view materialization works. Done means ephemeral materialization preserves the source fields and the compiled models no longer produce all-null records.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.