dbt-labs / dbt-labs/dbt-adapters

[Bug] Numeric type column reduction in size (e.g. `numeric(2) > numeric(1)`) treated as a schema change resulting in an error if using 'sync_all_columns'

Open
#985 0 comments 0 reactions 0 assignees View on GitHub
feature:incremental pkg:dbt-postgres pkg:dbt-snowflake type:bug
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this a new bug?

- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug

### Which packages are affected?

- [x] dbt-adapters
- [ ] dbt-tests-adapter
- [ ] dbt-athena
- [ ] dbt-athena-community
- [ ] dbt-bigquery
- [x] dbt-postgres
- [ ] dbt-redshift
- [x] dbt-snowflake
- [ ] dbt-spark

### Current Behavior

This is basically https://github.com/dbt-labs/dbt-core/issues/5351 but for numeric type. If the incoming data is of a smaller size (`numeric(1)`) compared to what is already in the target table (`numeric(2)`) - we run the on_schema_change alter DDL and attempt to reduce the size of the existing column which results in an error.

### Expected Behavior

This should be treated like the char decreasing in size scenario (`char(2) > char(1)`) - ignore if size is smaller than what is already in the target table.

### Steps To Reproduce

1. Project setup:

```yml
# dbt_project.yml
name: my_dbt_project
profile: all
version: "1.0.0"

models:
my_dbt_project:
+materialized: incremental
+on_schema_change: sync_all_columns
```

```sql
-- models/numeric_reduction.sql
select 11::numeric(2) as n

-- models/char_reduction.sql
select 'aa'::char(2) as c
```

2. Initial build:

```sh
$ dbt build --full-refresh

23:54:57 Running with dbt=1.10.0-b1
23:54:57 Registered adapter: postgres=1.9.0
23:54:57 Unable to do partial parsing because a project config has changed
23:54:57 Found 2 models, 433 macros
23:54:57
23:54:57 Concurrency: 4 threads (target='pg')
23:54:57
23:54:58 1 of 2 START sql incremental model public.char_reduction ....................... [RUN]
23:54:58 2 of 2 START sql incremental model public.numeric_reduction .................... [RUN]
23:54:58 1 of 2 OK created sql incremental model public.char_reduction .................. [SELECT 1 in 0.06s]
23:54:58 2 of 2 OK created sql incremental model public.numeric_reduction ............... [SELECT 1 in 0.06s]
23:54:58
23:54:58 Finished running 2 incremental models in 0 hours 0 minutes and 0.18 seconds (0.18s).
23:54:58
23:54:58 Completed successfully
23:54:58
23:54:58 Done. PASS=2 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=2
```

3. Reduce data size and rebuild:

```sql
-- models/numeric_reduction.sql
select 1::numeric(1) as n

-- models/char_reduction.sql
select 'a'::char(1) as c
```

```sh
$ dbt build
23:57:23 1 of 2 START sql incremental model public.char_reduction ....................... [RUN]
23:57:23 Re-using an available connection from the pool (formerly list_postgres_public, now model.my_dbt_project.char_reduction)
23:57:23 Began compiling node model.my_dbt_project.char_reduction
23:57:23 Writing injected SQL for node "model.my_dbt_project.char_reduction"
23:57:23 Began executing node model.my_dbt_project.char_reduction
23:57:23 Using postgres connection "model.my_dbt_project.char_reduction"
23:57:23 On model.my_dbt_project.char_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.char_reduction"} */



create temporary table "char_reduction__dbt_tmp115723564698"


as

(
select 'a'::char(1) as c
);


23:57:23 Opening a new connection, currently in state closed
23:57:23 SQL status: SELECT 1 in 0.010 seconds
23:57:23 Using postgres connection "model.my_dbt_project.char_reduction"
23:57:23 On model.my_dbt_project.char_reduction: BEGIN
23:57:23 SQL status: BEGIN in 0.001 seconds
23:57:23 Using postgres connection "model.my_dbt_project.char_reduction"
23:57:23 On model.my_dbt_project.char_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.char_reduction"} */

select
column_name,
data_type,
character_maximum_length,
numeric_precision,
numeric_scale

from INFORMATION_SCHEMA.columns
where table_name = 'char_reduction__dbt_tmp115723564698'

order by ordinal_position


23:57:23 SQL status: SELECT 1 in 0.006 seconds
23:57:23 Using postgres connection "model.my_dbt_project.char_reduction"
23:57:23 On model.my_dbt_project.char_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.char_reduction"} */

select
column_name,
data_type,
character_maximum_length,
numeric_precision,
numeric_scale

from "postgres".INFORMATION_SCHEMA.columns
where table_name = 'char_reduction'

and table_schema = 'public'

order by ordinal_position


23:57:23 SQL status: SELECT 1 in 0.002 seconds
23:57:23 Using postgres connection "model.my_dbt_project.char_reduction"
23:57:23 On model.my_dbt_project.char_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.char_reduction"} */

select
column_name,
data_type,
character_maximum_length,
numeric_precision,
numeric_scale

from INFORMATION_SCHEMA.columns
where table_name = 'char_reduction__dbt_tmp115723564698'

order by ordinal_position


23:57:23 SQL status: SELECT 1 in 0.001 seconds
23:57:23 Using postgres connection "model.my_dbt_project.char_reduction"
23:57:23 On model.my_dbt_project.char_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.char_reduction"} */

select
column_name,
data_type,
character_maximum_length,
numeric_precision,
numeric_scale

from "postgres".INFORMATION_SCHEMA.columns
where table_name = 'char_reduction'

and table_schema = 'public'

order by ordinal_position


23:57:23 SQL status: SELECT 1 in 0.003 seconds
23:57:23
In "postgres"."public"."char_reduction":
Schema changed: False
Source columns not in target: []
Target columns not in source: []
New column types: []

23:57:23 Writing runtime sql for node "model.my_dbt_project.char_reduction"
23:57:23 Using postgres connection "model.my_dbt_project.char_reduction"
23:57:23 On model.my_dbt_project.char_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.char_reduction"} */

insert into "postgres"."public"."char_reduction" ("c")
(
select "c"
from "char_reduction__dbt_tmp115723564698"
)


23:57:23 SQL status: INSERT 0 1 in 0.001 seconds
23:57:23 On model.my_dbt_project.char_reduction: COMMIT
23:57:23 Using postgres connection "model.my_dbt_project.char_reduction"
23:57:23 On model.my_dbt_project.char_reduction: COMMIT
23:57:23 SQL status: COMMIT in 0.002 seconds
23:57:23 On model.my_dbt_project.char_reduction: Close
23:57:23 Sending event: {'category': 'dbt', 'action': 'run_model', 'label': 'a9d5597e-209f-46fa-be12-a6da2a8d9ac3', 'context': []}
23:57:23 1 of 2 OK created sql incremental model public.char_reduction .................. [INSERT 0 1 in 0.09s]
...
23:57:23 2 of 2 START sql incremental model public.numeric_reduction .................... [RUN]
23:57:23 Re-using an available connection from the pool (formerly model.my_dbt_project.char_reduction, now model.my_dbt_project.numeric_reduction)
23:57:23 Began compiling node model.my_dbt_project.numeric_reduction
23:57:23 Writing injected SQL for node "model.my_dbt_project.numeric_reduction"
23:57:23 Began executing node model.my_dbt_project.numeric_reduction
23:57:23 Using postgres connection "model.my_dbt_project.numeric_reduction"
23:57:23 On model.my_dbt_project.numeric_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.numeric_reduction"} */



create temporary table "numeric_reduction__dbt_tmp115723646865"


as

(
select 1::numeric(1) as n
);


23:57:23 Opening a new connection, currently in state closed
23:57:23 SQL status: SELECT 1 in 0.009 seconds
23:57:23 Using postgres connection "model.my_dbt_project.numeric_reduction"
23:57:23 On model.my_dbt_project.numeric_reduction: BEGIN
23:57:23 SQL status: BEGIN in 0.000 seconds
23:57:23 Using postgres connection "model.my_dbt_project.numeric_reduction"
23:57:23 On model.my_dbt_project.numeric_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.numeric_reduction"} */

select
column_name,
data_type,
character_maximum_length,
numeric_precision,
numeric_scale

from INFORMATION_SCHEMA.columns
where table_name = 'numeric_reduction__dbt_tmp115723646865'

order by ordinal_position


23:57:23 SQL status: SELECT 1 in 0.003 seconds
23:57:23 Using postgres connection "model.my_dbt_project.numeric_reduction"
23:57:23 On model.my_dbt_project.numeric_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.numeric_reduction"} */

select
column_name,
data_type,
character_maximum_length,
numeric_precision,
numeric_scale

from "postgres".INFORMATION_SCHEMA.columns
where table_name = 'numeric_reduction'

and table_schema = 'public'

order by ordinal_position


23:57:23 SQL status: SELECT 1 in 0.002 seconds
23:57:23 Using postgres connection "model.my_dbt_project.numeric_reduction"
23:57:23 On model.my_dbt_project.numeric_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.numeric_reduction"} */

select
column_name,
data_type,
character_maximum_length,
numeric_precision,
numeric_scale

from INFORMATION_SCHEMA.columns
where table_name = 'numeric_reduction__dbt_tmp115723646865'

order by ordinal_position


23:57:23 SQL status: SELECT 1 in 0.001 seconds
23:57:23 Using postgres connection "model.my_dbt_project.numeric_reduction"
23:57:23 On model.my_dbt_project.numeric_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.numeric_reduction"} */

select
column_name,
data_type,
character_maximum_length,
numeric_precision,
numeric_scale

from "postgres".INFORMATION_SCHEMA.columns
where table_name = 'numeric_reduction'

and table_schema = 'public'

order by ordinal_position


23:57:23 SQL status: SELECT 1 in 0.001 seconds
23:57:23
In "postgres"."public"."numeric_reduction":
Schema changed: True
Source columns not in target: []
Target columns not in source: []
New column types: [{'column_name': 'n', 'new_type': 'numeric(1,0)'}]

23:57:23 Using postgres connection "model.my_dbt_project.numeric_reduction"
23:57:23 On model.my_dbt_project.numeric_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.numeric_reduction"} */

alter table "postgres"."public"."numeric_reduction" add column "n__dbt_alter" numeric(1,0);
update "postgres"."public"."numeric_reduction" set "n__dbt_alter" = "n";
alter table "postgres"."public"."numeric_reduction" drop column "n" cascade;
alter table "postgres"."public"."numeric_reduction" rename column "n__dbt_alter" to "n"

23:57:23 Postgres adapter: Postgres error: numeric field overflow
DETAIL: A field with precision 1, scale 0 must round to an absolute value less than 10^1.

23:57:23 On model.my_dbt_project.numeric_reduction: ROLLBACK
23:57:23 On model.my_dbt_project.numeric_reduction: Close
23:57:23 Database Error in model numeric_reduction (models/numeric_reduction.sql)
numeric field overflow
DETAIL: A field with precision 1, scale 0 must round to an absolute value less than 10^1.
23:57:23 Sending event: {'category': 'dbt', 'action': 'run_model', 'label': 'a9d5597e-209f-46fa-be12-a6da2a8d9ac3', 'context': []}
23:57:23 2 of 2 ERROR creating sql incremental model public.numeric_reduction ........... [ERROR in 0.04s]
```

^ Observe that we didn't attempt to reduce the size of the `char` column but we did the `numeric` column - leading to an error.

### Relevant log output

```shell
See above.
```

### Environment

```markdown
- OS: macOS
- Python: 3.11.9
- dbt-core: 1.10.0-b1
- dbt-adapters: 1.14.4
- dbt-postgres: 1.9.0
```

### Additional Context

Also tested this on Snowflake and it was the same behaviour.

```sql
-- models/numeric_reduction.sql
select 11 as n

-- models/char_reduction.sql
select 'aa' as c
```

^ Full refresh build then:

```sql
-- models/numeric_reduction.sql
select 1 as n

-- models/char_reduction.sql
select 'a' as c
```

```sh
$ dbt build
...
00:01:37 2 of 2 START sql incremental model sch.numeric_reduction ....................... [RUN]
00:01:37 Re-using an available connection from the pool (formerly model.my_dbt_project.char_reduction, now model.my_dbt_project.numeric_reduction)
00:01:37 Began compiling node model.my_dbt_project.numeric_reduction
00:01:37 Writing injected SQL for node "model.my_dbt_project.numeric_reduction"
00:01:37 Began executing node model.my_dbt_project.numeric_reduction
00:01:37 Using snowflake connection "model.my_dbt_project.numeric_reduction"
00:01:37 On model.my_dbt_project.numeric_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "sf", "node_id": "model.my_dbt_project.numeric_reduction"} */
create or replace temporary view db.sch.numeric_reduction__dbt_tmp

as (
select 1 as n
);
00:01:37 SQL status: SUCCESS 1 in 0.582 seconds
00:01:37 Using snowflake connection "model.my_dbt_project.numeric_reduction"
00:01:37 On model.my_dbt_project.numeric_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "sf", "node_id": "model.my_dbt_project.numeric_reduction"} */
describe table db.sch.numeric_reduction__dbt_tmp
00:01:38 SQL status: SUCCESS 1 in 0.401 seconds
00:01:38 Using snowflake connection "model.my_dbt_project.numeric_reduction"
00:01:38 On model.my_dbt_project.numeric_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "sf", "node_id": "model.my_dbt_project.numeric_reduction"} */
describe table db.sch.numeric_reduction
00:01:38 SQL status: SUCCESS 1 in 0.381 seconds
00:01:38 Using snowflake connection "model.my_dbt_project.numeric_reduction"
00:01:38 On model.my_dbt_project.numeric_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "sf", "node_id": "model.my_dbt_project.numeric_reduction"} */
describe table db.sch.numeric_reduction__dbt_tmp
00:01:39 SQL status: SUCCESS 1 in 0.389 seconds
00:01:39 Using snowflake connection "model.my_dbt_project.numeric_reduction"
00:01:39 On model.my_dbt_project.numeric_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "sf", "node_id": "model.my_dbt_project.numeric_reduction"} */
describe table "DB"."SCH"."NUMERIC_REDUCTION"
00:01:39 SQL status: SUCCESS 1 in 0.354 seconds
00:01:39
In "DB"."SCH"."NUMERIC_REDUCTION":
Schema changed: True
Source columns not in target: []
Target columns not in source: []
New column types: [{'column_name': 'N', 'new_type': 'NUMBER(1,0)'}]

00:01:39 Using snowflake connection "model.my_dbt_project.numeric_reduction"
00:01:39 On model.my_dbt_project.numeric_reduction: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "sf", "node_id": "model.my_dbt_project.numeric_reduction"} */
alter table "DB"."SCH"."NUMERIC_REDUCTION" alter "N" set data type NUMBER(1,0);
00:01:40 Snowflake adapter: Snowflake query id: 01bb9581-0708-0899-000d-378347de613a
00:01:40 Snowflake adapter: Snowflake error: 040051 (22000): SQL compilation error: cannot change column N from type NUMBER(2,0) to NUMBER(1,0) because some existing values cannot be represented using precision 1 instead of precision 2.

00:01:40 Database Error in model numeric_reduction (models/numeric_reduction.sql)
040051 (22000): SQL compilation error: cannot change column N from type NUMBER(2,0) to NUMBER(1,0) because some existing values cannot be represented using precision 1 instead of precision 2.

00:01:40 Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '5ddf7499-1f80-457b-a4d9-3f93dec15112', 'context': []}
00:01:40 2 of 2 ERROR creating sql incremental model sch.numeric_reduction .............. [ERROR in 2.67s]
```

On Snowflake, it auto detected the size `number(2,0)` and didn't need to cast it like I did in my testing above for postgres. However, the behaviour is basically identical, a decreased number type column is not ignored unlike a decreasing char type column.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.