aws-samples / aws-samples/dbt-glue

Support for Data Type Changes in Iceberg

Open
#533 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

In many cases, changing data types doesn’t result in data loss. Where Iceberg supports such changes, `dbt-glue` should accommodate them.

This can be achieved by maintaining a **dictionary** mapping which data type changes are allowed. For instance:

```python
supported_type_changes = {
"int": ["bigint", "float"],
"float": ["double"],
"string": ["text", "varchar"]
}
```

A second phase, this could involve implementing a add/drop logic to handle more complex data type changes based on user preferences (even if data is lost). I can provide a detailed specification for this if needed.

Contributor guide

Open the contributing guide

Research direction

The issue does not name any files, tests, or entry points. Start by locating the adapter's existing Iceberg schema-change handling and identify how current and target data types are represented. Done means supported non-lossy type changes are accepted through a defined compatibility mapping; the proposed add/drop behavior is described only as a possible later phase.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.