dlt-hub / dlt-hub/dlt

feat(fabric): TINYINT source columns emit unsupported `tinyint` DDL and fail the load step

Open
#4,306 0 comments 0 reactions 1 assignee Claimed by @anuunchin View on GitHub
destination enhancement
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### Problem

Fabric Warehouse has no `tinyint`. Its narrowest integer type is `smallint`. The Fabric destination emits `tinyint` anyway, so `CREATE TABLE` fails:

```
dlt.destinations.exceptions.DatabaseTerminalException: Driver Error: Syntax error or access violation;
[Microsoft][SQL Server]The data type 'tinyint' in column 'foo' is not supported in this edition of SQL Server.
```

Because it is a `DatabaseTerminalException` the load is not retried and the package stays pending, so every following run of that pipeline fails on the same table. A single `TINYINT` column anywhere in a source schema blocks the whole schema from loading.

### Root cause

`sql_database` reflects `TINYINT` as dlt type `bigint` with `precision=8`. On the way out, `MsSqlTypeMapper.to_db_integer_type` maps that precision to the narrowest SQL Server integer type:

```python
# dlt/destinations/impl/mssql/factory.py
if precision <= 8:
return "tinyint"
```

That is correct for SQL Server and Synapse but not for Fabric. `FabricTypeMapper` inherits from `SynapseTypeMapper`, which inherits from `MsSqlTypeMapper`, and currently overrides only the datetime and string mappings -- so the `tinyint` branch is reached unchanged.

Verified against `devel`:

```
precision=8 -> tinyint
precision=16 -> smallint
precision=32 -> int
precision=64 -> bigint
```

This is the same class of problem as the already-handled `datetimeoffset` -> `datetime2` and `nvarchar` -> `varchar` mismatches: an inherited SQL Server type that Fabric Warehouse rejects.

### Reproduction

1. Create a SQL Server source table with a `TINYINT` column:

```sql
CREATE TABLE dbo.Foo (Id int NOT NULL, Bar tinyint NOT NULL);
```

2. Run a `sql_database` source with `reflection_level="full_with_precision"` into a `fabric` destination.
3. Extract and normalize succeed; load fails on `CREATE TABLE`.

### Current workaround

Use `reflection_level="full"` so no precision is reflected and the column becomes `bigint`, losing the width information for every integer column in the schema.

### Proposed fix

Override `to_db_integer_type` in `FabricTypeMapper` so any precision at or below 8 widens to `smallint`, then delegate to the parent. Widening is safe: `TINYINT` values (0-255) all fit in `smallint`.

A test asserting that a `bigint` column with `precision=8` yields `smallint` on the Fabric destination would guard the inheritance chain, which is where this class of regression keeps appearing.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.