dlt-hub / dlt-hub/dlt

`primary_key` not removed with `apply_hints(primary_key="")` (same issue with `merge_key`)

Open
#3,210 1 comment 0 reactions 1 assignee Claimed by @anuunchin View on GitHub
bug
Dominant language
Python
Stars
5.9k
Forks
605
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### dlt version

1.17.1

### Describe the problem

Providing an empty value for `primary_key` to `apply_hints` does not remove a primary key that was set before:

1. `my_resource.apply_hints(primary_key="id")` sets the primary key correctly
2. `my_resource.apply_hints(primary_key="")` does not remove the primary key (neither does `my_resource.apply_hints(primary_key=[])`)

The documentation for `apply_hints` says:

_"Pass empty value (for a particular type i.e. "" for a string) to remove a hint."_

The same behavior occurs for `merge_key`.

I found this related issue that may suggest this is intended behavior, but I'm not sure and I think this behavior is confusing: https://github.com/dlt-hub/dlt/issues/2440

### Expected behavior

`my_resource.apply_hints(primary_key="")` (or `my_resource.apply_hints(primary_key=[])` removes the primary key (if exists)

### Steps to reproduce

```py
import json

import dlt

@dlt.resource
def my_table():
return [{"id": 1, "foo": 1}, {"id": 2, "foo": 1}]

pipe = dlt.pipeline(destination="duckdb")

# Set primary_key using apply_hints
my_table.apply_hints(primary_key="id")
pipe.run(my_table())
table_schema = pipe.default_schema.get_table("my_table")
print('Table schema after `my_table.apply_hints(primary_key="id")` (primary_key set: CORRECT):')
print(json.dumps(table_schema, indent=2))

# Attempt to unset primary_key through apply_hints: doesn't work
my_table.apply_hints(primary_key="") # primary_key=[] has same behavior as primary_key=""
pipe.run(my_table())
table_schema = pipe.default_schema.get_table("my_table")
print()
print('Table schema after `my_table.apply_hints(primary_key="")` (primary_key still set: INCORRECT):')
print(json.dumps(table_schema, indent=2))

# Set primary_key to False using `columns` as workaround: works
my_table.apply_hints(columns={"id": {"primary_key": False}})
pipe.run(my_table())
table_schema = pipe.default_schema.get_table("my_table")
print()
print('Table schema after `my_table.apply_hints(columns={"id": {"primary_key": False}})` (primary_key set to False: CORRECT, but workaround):')
print(json.dumps(table_schema, indent=2))
```

Output:

```sh
Table schema after `my_table.apply_hints(primary_key="id")` (primary_key set: CORRECT):
{
"columns": {
"id": {
"name": "id",
"nullable": false,
"primary_key": true, # CORRECT
"data_type": "bigint"
},
"foo": {
"name": "foo",
"data_type": "bigint",
"nullable": true
},
"_dlt_load_id": {
"name": "_dlt_load_id",
"data_type": "text",
"nullable": false
},
"_dlt_id": {
"name": "_dlt_id",
"data_type": "text",
"nullable": false,
"unique": true,
"row_key": true
}
},
"write_disposition": "append",
"name": "my_table",
"resource": "my_table",
"x-normalizer": {
"seen-data": true
}
}

Table schema after `my_table.apply_hints(primary_key="")` (primary_key still set: INCORRECT):
{
"columns": {
"id": {
"name": "id",
"nullable": false,
"primary_key": true, # INCORRECT
"data_type": "bigint"
},
"foo": {
"name": "foo",
"data_type": "bigint",
"nullable": true
},
"_dlt_load_id": {
"name": "_dlt_load_id",
"data_type": "text",
"nullable": false
},
"_dlt_id": {
"name": "_dlt_id",
"data_type": "text",
"nullable": false,
"unique": true,
"row_key": true
}
},
"write_disposition": "append",
"name": "my_table",
"resource": "my_table",
"x-normalizer": {
"seen-data": true
}
}

Table schema after `my_table.apply_hints(columns={"id": {"primary_key": False}})` (primary_key set to False: CORRECT, but workaround):
{
"columns": {
"id": {
"name": "id",
"nullable": false,
"primary_key": false, # CORRECT
"data_type": "bigint"
},
"foo": {
"name": "foo",
"data_type": "bigint",
"nullable": true
},
"_dlt_load_id": {
"name": "_dlt_load_id",
"data_type": "text",
"nullable": false
},
"_dlt_id": {
"name": "_dlt_id",
"data_type": "text",
"nullable": false,
"unique": true,
"row_key": true
}
},
"write_disposition": "append",
"name": "my_table",
"resource": "my_table",
"x-normalizer": {
"seen-data": true
}
}
````

### Operating system

Linux

### Runtime environment

Local

### Python version

3.13

### dlt data source

_No response_

### dlt destination

_No response_

### Other deployment details

_No response_

### Additional information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.