dbt-labs / dbt-labs/dbt-adapters

[Bug] Error refreshing Redshift materialized view without full refresh when view has no data

Open
#1,256 7 comments 1 reaction 0 assignees View on GitHub
pkg:dbt-redshift type:bug
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this a new bug?

- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug

### Which packages are affected?

- [ ] dbt-adapters
- [ ] dbt-tests-adapter
- [ ] dbt-athena
- [ ] dbt-athena-community
- [ ] dbt-bigquery
- [ ] dbt-postgres
- [x] dbt-redshift
- [ ] dbt-snowflake
- [ ] dbt-spark

### Current Behavior

I have a materialized view generated by dbt that sometimes contains no data (where there is insufficient data from the underlying tables). Currently, this model is specified with a full refresh with every run.

When trying to execute `dbt run` without a full refresh specified via config or via `--full-refresh`, an error is thrown in the case where the materialized view exists but contains no rows of data.

The same error is not thrown when a `full_refresh` is specified.

```
Compilation Error in model (models/staging/enterprise/.sql)
RedshiftMaterializedViewConfig.__init__() missing 3 required positional arguments: 'mv_name', 'schema_name', and 'database_name'

> in macro get_materialized_view_configuration_changes (macros/relations/materialized_view/alter.sql)
> called by macro materialized_view_get_build_sql (macros/materializations/models/materialized_view.sql)
> called by macro materialization_materialized_view_default (macros/materializations/models/materialized_view.sql)
> called by model (models/staging/enterprise/.sql)
```

### Expected Behavior

It should be possible to refresh materialized views that have no rows of data when full refresh is not specified.

### Steps To Reproduce

1. Create a test schema and test table on redshift that dbt user can read, write and create materialized views for.
```
CRAETE SCHEMA test_schema;

CREATE TABLE test_schema.test_table (
id INT PRIMARY KEY,
name VARCHAR(255),
created_at TIMESTAMP
);

INSERT INTO test_schema.test_table (id, name)
VALUES (1, 'test' );
```

3. Create a dbt project and model that creates a materialized view on Redshift that returns no rows of data from the source table.

`example.sql`:
```
{{ config(materialized='materialized_view') }}

select id,
name
from {{ source('test_source', 'test_table') }} where id > 2

```

`__sources.yml`
```
sources:
- database:
description: ' test_source'
name: test_source
schema: test_schema
tables:
- description: test_table
name: test_table
version: 2

```

4. Execute `dbt run --full-refresh`.
5. Execute `dbt run` without full refresh.
6. Observe the error.
7. It may be that this query does not return a matching result when the underlying table contains no data:
```
select
tb.database,
tb.schema,
tb.table,
tb.diststyle,
tb.sortkey1,
mv.autorefresh
from svv_table_info tb
-- svv_mv_info is queryable by Redshift Serverless, but stv_mv_info is not
left join svv_mv_info mv
on mv.database_name = tb.database
and mv.schema_name = tb.schema
and mv.name = tb.table
where tb.table ilike 'example'
and tb.schema ilike ''
and tb.database ilike '';

```

### Relevant log output

```shell
11:30:05.673131 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [, , ]}

============================== 11:30:05.680441 | 156b942d-d82e-4f20-a14d-dc613334b885 ==============================
11:30:05.680441 [info ] [MainThread]: Running with dbt=1.10.7
11:30:05.680742 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'profiles_dir': '~/Sites/issues//dbt-case/dbt_case', 'version_check': 'True', 'warn_error': 'None', 'log_path': '~/Sites/issues//dbt-case/dbt_case/logs', 'fail_fast': 'False', 'debug': 'False', 'use_colors': 'True', 'use_experimental_parser': 'False', 'empty': 'False', 'quiet': 'False', 'no_print': 'None', 'log_format': 'default', 'static_parser': 'True', 'invocation_command': 'dbt run --target dev', 'introspect': 'True', 'target_path': 'None', 'warn_error_options': 'WarnErrorOptionsV2(error=[], warn=[], silence=[])', 'send_anonymous_usage_stats': 'True'}
11:30:05.829458 [debug] [MainThread]: Redshift adapter: Setting redshift_connector to ERROR
11:30:05.829727 [debug] [MainThread]: Redshift adapter: Setting redshift_connector.core to ERROR
11:30:05.930280 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '156b942d-d82e-4f20-a14d-dc613334b885', 'context': []}
11:30:05.964248 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': '156b942d-d82e-4f20-a14d-dc613334b885', 'context': []}
11:30:05.965297 [info ] [MainThread]: Registered adapter: redshift=1.9.5
11:30:06.015993 [debug] [MainThread]: checksum: 0e8ea0674924c247293f8e885046b8f8529d3d13cd06aff2725aca579a7f383e, vars: {}, profile: , target: dev, version: 1.10.7
11:30:06.100871 [debug] [MainThread]: Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
11:30:06.101211 [debug] [MainThread]: Partial parsing enabled, no changes found, skipping parsing
11:30:06.127215 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'load_project', 'label': '156b942d-d82e-4f20-a14d-dc613334b885', 'context': []}
11:30:06.171980 [debug] [MainThread]: Wrote artifact WritableManifest to ~/Sites/issues//dbt-case/dbt_case/target/manifest.json
11:30:06.173517 [debug] [MainThread]: Wrote artifact SemanticManifest to ~/Sites/issues//dbt-case/dbt_case/target/semantic_manifest.json
11:30:06.183833 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'resource_counts', 'label': '156b942d-d82e-4f20-a14d-dc613334b885', 'context': []}
11:30:06.184108 [info ] [MainThread]: Found 1 model, 1 source, 505 macros
11:30:06.184296 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '156b942d-d82e-4f20-a14d-dc613334b885', 'context': []}
11:30:06.186989 [info ] [MainThread]:
11:30:06.187236 [info ] [MainThread]: Concurrency: 1 threads (target='dev')
11:30:06.187387 [info ] [MainThread]:
11:30:06.187791 [debug] [MainThread]: Acquiring new redshift connection 'master'
11:30:06.188963 [debug] [ThreadPool]: Acquiring new redshift connection 'list_'
11:30:06.206251 [debug] [ThreadPool]: Using redshift connection "list_"
11:30:06.206472 [debug] [ThreadPool]: On list_: /* {"app": "dbt", "dbt_version": "1.10.7", "profile_name": "dbt_case", "target_name": "dev", "connection_name": "list_"} */

select distinct nspname from pg_namespace
11:30:06.206632 [debug] [ThreadPool]: Opening a new connection, currently in state init
11:30:06.206867 [debug] [ThreadPool]: Redshift adapter: Connecting to Redshift with 'database' credentials method
11:30:06.209008 [debug] [ThreadPool]: Redshift adapter: Establishing connection using ssl with `sslmode` set to 'prefer'.To connect without ssl, set `sslmode` to 'disable'.
11:30:06.501159 [debug] [ThreadPool]: SQL status: SUCCESS in 0.294 seconds
11:30:06.583528 [debug] [ThreadPool]: On list_: Close
11:30:06.584659 [debug] [ThreadPool]: Re-using an available connection from the pool (formerly list_, now list__)
11:30:06.588744 [debug] [ThreadPool]: Using redshift connection "list__"
11:30:06.588952 [debug] [ThreadPool]: On list__: BEGIN
11:30:06.589108 [debug] [ThreadPool]: Opening a new connection, currently in state closed
11:30:06.589258 [debug] [ThreadPool]: Redshift adapter: Connecting to Redshift with 'database' credentials method
11:30:06.589461 [debug] [ThreadPool]: Redshift adapter: Establishing connection using ssl with `sslmode` set to 'prefer'.To connect without ssl, set `sslmode` to 'disable'.
11:30:06.839451 [debug] [ThreadPool]: SQL status: SUCCESS in 0.250 seconds
11:30:06.840062 [debug] [ThreadPool]: Using redshift connection "list__"
11:30:06.840486 [debug] [ThreadPool]: On list__: /* {"app": "dbt", "dbt_version": "1.10.7", "profile_name": "dbt_case", "target_name": "dev", "connection_name": "list__"} */
select
table_catalog as database,
table_name as name,
table_schema as schema,
'table' as type
from information_schema.tables
where table_schema ilike ''
and table_type = 'BASE TABLE'
union all
select
table_catalog as database,
table_name as name,
table_schema as schema,
case
when view_definition ilike '%create materialized view%'
then 'materialized_view'
else 'view'
end as type
from information_schema.views
where table_schema ilike ''
11:30:06.936664 [debug] [ThreadPool]: SQL status: SUCCESS in 0.095 seconds
11:30:06.941128 [debug] [ThreadPool]: On list__: ROLLBACK
11:30:07.050526 [debug] [ThreadPool]: On list__: Close
11:30:07.066214 [debug] [MainThread]: Using redshift connection "master"
11:30:07.066672 [debug] [MainThread]: On master: BEGIN
11:30:07.066987 [debug] [MainThread]: Opening a new connection, currently in state init
11:30:07.067284 [debug] [MainThread]: Redshift adapter: Connecting to Redshift with 'database' credentials method
11:30:07.067688 [debug] [MainThread]: Redshift adapter: Establishing connection using ssl with `sslmode` set to 'prefer'.To connect without ssl, set `sslmode` to 'disable'.
11:30:07.324833 [debug] [MainThread]: SQL status: SUCCESS in 0.257 seconds
11:30:07.326489 [debug] [MainThread]: Using redshift connection "master"
11:30:07.327494 [debug] [MainThread]: On master: /* {"app": "dbt", "dbt_version": "1.10.7", "profile_name": "dbt_case", "target_name": "dev", "connection_name": "master"} */
with
relation as (
select
pg_class.oid as relation_id,
pg_class.relname as relation_name,
pg_class.relnamespace as schema_id,
pg_namespace.nspname as schema_name,
pg_class.relkind as relation_type
from pg_class
join pg_namespace
on pg_class.relnamespace = pg_namespace.oid
where pg_namespace.nspname != 'information_schema'
and pg_namespace.nspname not like 'pg\_%'
),
dependency as (
select distinct
coalesce(pg_rewrite.ev_class, pg_depend.objid) as dep_relation_id,
pg_depend.refobjid as ref_relation_id,
pg_depend.refclassid as ref_class_id
from pg_depend
left join pg_rewrite
on pg_depend.objid = pg_rewrite.oid
where coalesce(pg_rewrite.ev_class, pg_depend.objid) != pg_depend.refobjid
)

select distinct
dep.schema_name as dependent_schema,
dep.relation_name as dependent_name,
ref.schema_name as referenced_schema,
ref.relation_name as referenced_name
from dependency
join relation ref
on dependency.ref_relation_id = ref.relation_id
join relation dep
on dependency.dep_relation_id = dep.relation_id
11:30:07.519884 [debug] [MainThread]: SQL status: SUCCESS in 0.191 seconds
11:30:07.531159 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '156b942d-d82e-4f20-a14d-dc613334b885', 'context': []}
11:30:07.532069 [debug] [MainThread]: On master: ROLLBACK
11:30:07.643624 [debug] [MainThread]: Using redshift connection "master"
11:30:07.644973 [debug] [MainThread]: On master: BEGIN
11:30:07.685490 [debug] [MainThread]: SQL status: SUCCESS in 0.039 seconds
11:30:07.686800 [debug] [MainThread]: On master: COMMIT
11:30:07.688472 [debug] [MainThread]: Using redshift connection "master"
11:30:07.689314 [debug] [MainThread]: On master: COMMIT
11:30:07.729471 [debug] [MainThread]: SQL status: SUCCESS in 0.039 seconds
11:30:07.730553 [debug] [MainThread]: On master: Close
11:30:07.742152 [debug] [Thread-6 (]: Began running node model.dbt_case.example
11:30:07.743431 [info ] [Thread-6 (]: 1 of 1 START sql materialized_view model .example ..................... [RUN]
11:30:07.744170 [debug] [Thread-6 (]: Re-using an available connection from the pool (formerly list__, now model.dbt_case.example)
11:30:07.744541 [debug] [Thread-6 (]: Began compiling node model.dbt_case.example
11:30:07.752491 [debug] [Thread-6 (]: Writing injected SQL for node "model.dbt_case.example"
11:30:07.753661 [debug] [Thread-6 (]: Began executing node model.dbt_case.example
11:30:07.797124 [debug] [Thread-6 (]: Determining configuration changes on: "".""."example"
11:30:07.804336 [debug] [Thread-6 (]: Using redshift connection "model.dbt_case.example"
11:30:07.804566 [debug] [Thread-6 (]: On model.dbt_case.example: /* {"app": "dbt", "dbt_version": "1.10.7", "profile_name": "dbt_case", "target_name": "dev", "node_id": "model.dbt_case.example"} */

select
tb.database,
tb.schema,
tb.table,
tb.diststyle,
tb.sortkey1,
mv.autorefresh
from svv_table_info tb
-- svv_mv_info is queryable by Redshift Serverless, but stv_mv_info is not
left join svv_mv_info mv
on mv.database_name = tb.database
and mv.schema_name = tb.schema
and mv.name = tb.table
where tb.table ilike 'example'
and tb.schema ilike ''
and tb.database ilike ''
11:30:07.804754 [debug] [Thread-6 (]: Opening a new connection, currently in state closed
11:30:07.804941 [debug] [Thread-6 (]: Redshift adapter: Connecting to Redshift with 'database' credentials method
11:30:07.805205 [debug] [Thread-6 (]: Redshift adapter: Establishing connection using ssl with `sslmode` set to 'prefer'.To connect without ssl, set `sslmode` to 'disable'.
11:30:09.376201 [debug] [Thread-6 (]: SQL status: SUCCESS in 1.569 seconds
11:30:09.384015 [debug] [Thread-6 (]: Using redshift connection "model.dbt_case.example"
11:30:09.384810 [debug] [Thread-6 (]: On model.dbt_case.example: /* {"app": "dbt", "dbt_version": "1.10.7", "profile_name": "dbt_case", "target_name": "dev", "node_id": "model.dbt_case.example"} */

SELECT
a.attname as column,
a.attisdistkey as is_dist_key,
a.attsortkeyord as sort_key_position
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
WHERE
n.nspname ilike ''
AND c.relname LIKE 'mv_tbl__example__%'
11:30:09.508584 [debug] [Thread-6 (]: SQL status: SUCCESS in 0.123 seconds
11:30:09.515534 [debug] [Thread-6 (]: Using redshift connection "model.dbt_case.example"
11:30:09.516102 [debug] [Thread-6 (]: On model.dbt_case.example: /* {"app": "dbt", "dbt_version": "1.10.7", "profile_name": "dbt_case", "target_name": "dev", "node_id": "model.dbt_case.example"} */

select
vw.definition
from pg_views vw
where vw.viewname = 'example'
and vw.schemaname = ''
and vw.definition ilike '%create materialized view%'
11:30:09.564712 [debug] [Thread-6 (]: SQL status: SUCCESS in 0.048 seconds
11:30:09.570195 [debug] [Thread-6 (]: On model.dbt_case.example: Close
11:30:09.582211 [debug] [Thread-6 (]: Compilation Error in model example (models/example/example.sql)
RedshiftMaterializedViewConfig.__init__() missing 3 required positional arguments: 'mv_name', 'schema_name', and 'database_name'

> in macro get_materialized_view_configuration_changes (macros/relations/materialized_view/alter.sql)
> called by macro materialized_view_get_build_sql (macros/materializations/models/materialized_view.sql)
> called by macro materialization_materialized_view_default (macros/materializations/models/materialized_view.sql)
> called by model example (models/example/example.sql)
11:30:09.585125 [debug] [Thread-6 (]: Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '156b942d-d82e-4f20-a14d-dc613334b885', 'context': []}
11:30:09.585851 [error] [Thread-6 (]: 1 of 1 ERROR creating sql materialized_view model .example ............ [ERROR in 1.84s]
11:30:09.586484 [debug] [Thread-6 (]: Finished running node model.dbt_case.example
11:30:09.587198 [debug] [Thread-9 (]: Marking all children of 'model.dbt_case.example' to be skipped because of status 'error'. Reason: Compilation Error in model example (models/example/example.sql)
RedshiftMaterializedViewConfig.__init__() missing 3 required positional arguments: 'mv_name', 'schema_name', and 'database_name'

> in macro get_materialized_view_configuration_changes (macros/relations/materialized_view/alter.sql)
> called by macro materialized_view_get_build_sql (macros/materializations/models/materialized_view.sql)
> called by macro materialization_materialized_view_default (macros/materializations/models/materialized_view.sql)
> called by model example (models/example/example.sql).
11:30:09.591180 [debug] [MainThread]: Using redshift connection "master"
11:30:09.591556 [debug] [MainThread]: On master: BEGIN
11:30:09.591795 [debug] [MainThread]: Opening a new connection, currently in state closed
11:30:09.592142 [debug] [MainThread]: Redshift adapter: Connecting to Redshift with 'database' credentials method
11:30:09.592655 [debug] [MainThread]: Redshift adapter: Establishing connection using ssl with `sslmode` set to 'prefer'.To connect without ssl, set `sslmode` to 'disable'.
11:30:09.841167 [debug] [MainThread]: SQL status: SUCCESS in 0.249 seconds
11:30:09.842893 [debug] [MainThread]: On master: COMMIT
11:30:09.844186 [debug] [MainThread]: Using redshift connection "master"
11:30:09.844765 [debug] [MainThread]: On master: COMMIT
11:30:09.885090 [debug] [MainThread]: SQL status: SUCCESS in 0.040 seconds
11:30:09.886348 [debug] [MainThread]: On master: Close
11:30:09.888506 [debug] [MainThread]: Connection 'master' was properly closed.
11:30:09.889061 [debug] [MainThread]: Connection 'model.dbt_case.example' was properly closed.
11:30:09.889754 [info ] [MainThread]:
11:30:09.890358 [info ] [MainThread]: Finished running 1 materialized view model in 0 hours 0 minutes and 3.70 seconds (3.70s).
11:30:09.891712 [debug] [MainThread]: Command end result
11:30:09.913636 [debug] [MainThread]: Wrote artifact WritableManifest to ~/Sites/issues//dbt-case/dbt_case/target/manifest.json
11:30:09.915116 [debug] [MainThread]: Wrote artifact SemanticManifest to ~/Sites/issues//dbt-case/dbt_case/target/semantic_manifest.json
11:30:09.920857 [debug] [MainThread]: Wrote artifact RunExecutionResult to ~/Sites/issues//dbt-case/dbt_case/target/run_results.json
11:30:09.921168 [info ] [MainThread]:
11:30:09.921484 [info ] [MainThread]: Completed with 1 error, 0 partial successes, and 0 warnings:
11:30:09.921763 [info ] [MainThread]:
11:30:09.922044 [error] [MainThread]: Failure in model example (models/example/example.sql)
11:30:09.922312 [error] [MainThread]: Compilation Error in model example (models/example/example.sql)
RedshiftMaterializedViewConfig.__init__() missing 3 required positional arguments: 'mv_name', 'schema_name', and 'database_name'

> in macro get_materialized_view_configuration_changes (macros/relations/materialized_view/alter.sql)
> called by macro materialized_view_get_build_sql (macros/materializations/models/materialized_view.sql)
> called by macro materialization_materialized_view_default (macros/materializations/models/materialized_view.sql)
> called by model example (models/example/example.sql)
11:30:09.922519 [info ] [MainThread]:
11:30:09.922733 [info ] [MainThread]: compiled code at target/compiled/dbt_case/models/example/example.sql
11:30:09.922976 [info ] [MainThread]:
11:30:09.923194 [info ] [MainThread]: Done. PASS=0 WARN=0 ERROR=1 SKIP=0 NO-OP=0 TOTAL=1
11:30:09.925473 [debug] [MainThread]: Resource report: {"command_name": "run", "command_success": false, "command_wall_clock_time": 4.2770824, "process_in_blocks": "0", "process_kernel_time": 0.285398, "process_mem_max_rss": "178339840", "process_out_blocks": "0", "process_user_time": 1.815124}
11:30:09.925824 [debug] [MainThread]: Command `dbt run` failed at 11:30:09.925762 after 4.28 seconds
11:30:09.926243 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [, , ]}
11:30:09.926481 [debug] [MainThread]: Flushing usage events
11:30:10.301161 [debug] [MainThread]: An error was encountered while trying to flush usage events
```

### Environment

```markdown
- OS: macOS Sequoia
- Python: 3.13.5
- dbt-adapters: 1.16.3
- dbt-redshift: 1.9.5
```

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.