aws-samples / aws-samples/dbt-glue
Support for Data Type Changes in Iceberg
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
In many cases, changing data types doesn’t result in data loss. Where Iceberg supports such changes, `dbt-glue` should accommodate them.
This can be achieved by maintaining a **dictionary** mapping which data type changes are allowed. For instance:
```python
supported_type_changes = {
"int": ["bigint", "float"],
"float": ["double"],
"string": ["text", "varchar"]
}
```
A second phase, this could involve implementing a add/drop logic to handle more complex data type changes based on user preferences (even if data is lost). I can provide a detailed specification for this if needed.
Contributor guide
Research direction
The issue does not name any files, tests, or entry points. Start by locating the adapter's existing Iceberg schema-change handling and identify how current and target data types are represented. Done means supported non-lossy type changes are accepted through a defined compatibility mapping; the proposed add/drop behavior is described only as a possible later phase.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100