dlt-hub / dlt-hub/dlt

SQL destinations should not add new required dlt internal columns as NOT NULL on existing non-empty nested tables

Open
#4,041 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
605
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### dlt version

1.27.2

### Describe the problem

A dlt pipeline loading deeply nested MongoDB data with merge disposition into Microsoft SQL Server fails during the load schema-update phase. This is an old problem not exclusive to this version.

The failing table is a generated nested child table several levels below the root resource. dlt attempts to add the internal root-key column _dlt_root_id as NOT NULL to an existing non-empty child table.

MSSQL rejects this DDL because SQL Server does not allow adding a NOT NULL column to a non-empty table unless a valid DEFAULT is provided or the table is empty.

Example failing migration pattern:

```
ALTER TABLE [schema].[nested_child_table]
ADD [_dlt_root_id] nvarchar(...) NOT NULL;
```

This fails with:

```
Column '_dlt_root_id' cannot be added to non-empty table ''
because it does not satisfy these conditions. (4901)
```

Funny enough, there as warning this will happen that I suppose is useful on scenarios where this is not caused by a permanent merge pipeline but by a change from replace to merge:
```
Column(s) [...] with NOT NULL are being added to existing table .
If there's data in the table the operation will fail.
```

### Expected behavior

Race condition or timing issue if found that causes the schema problem fixed, or:
For existing non-empty tables, SQL destinations should not emit ALTER TABLE ADD NOT NULL unless they also provide a valid default, perform a backfill first, or verify that the table is empty.

For MSSQL, dlt should add newly evolved required columns as nullable first:
```
ALTER TABLE [schema].[nested_child_table]
ADD [_dlt_root_id] nvarchar(1000) NULL;
```
If dlt can safely backfill valid values later, it can then enforce NOT NULL.

### Steps to reproduce

The exact reason why _dlt_root_id is added after the table already contains data is unknown.

A suspicious code path is in the relational JSON normalizer: _normalize_row() only propagates root-key values when self.propagation_config already exists, while extend_table() can create that propagation config later when requires_root_key() is true. In a complex schema-evolution path, a deeply nested table may be created before _dlt_root_id is consistently propagated, then later dlt tries to add _dlt_root_id as required.

This timing hypothesis is not proven. A simple clean nested merge reproduction creates _dlt_root_id correctly on the first run. The verified problem is the destination schema migration: dlt emits invalid DDL for SQL Server when evolving an existing non-empty table.

Possible code backed causes for the table schema problem (found in colab with AI at least):
- dynamic schema/table discovery during normalization
- propagation config missing when the normalizer instance starts
- extend_table() adding propagation after a parent row has already been yielded
- deep nested MongoDB schema evolution
- persisted or pending package schema state

### Operating system

Linux

### Runtime environment

Docker, Docker Compose

### Python version

3.13

### dlt data source

Mongodb

### dlt destination

_No response_

### Other deployment details

_No response_

### Additional information

This is not necessarily MSSQL-only. Any destination adapter that emits ALTER TABLE ADD NOT NULL for an existing non-empty table can fail similarly.

Likely affected classes:

MSSQL: confirmed failure, SQL Server rejects this DDL.
Snowflake: should fail similarly if dlt emits ADD COLUMN ... NOT NULL on a non-empty table without a default.
BigQuery: should fail similarly if dlt attempts to add a new REQUIRED field to an existing table schema.
Postgres: adding a nullable column works, but adding/enforcing NOT NULL on existing rows requires valid data/default/backfill.

Contributor guide

Open the contributing guide

Research direction

Trace the relational JSON normalizer entry points _normalize_row(), extend_table(), and requires_root_key(), then follow the destination schema-migration path that emits ALTER TABLE statements. Reproduce the evolution case with an existing non-empty nested table if possible. Done means required internal columns are not added with invalid NOT NULL DDL, or are safely backfilled before enforcement, with coverage for the affected SQL destinations.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.