Propagate Column Descriptions from DLT Schema to Redshift
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 38
Description
### Feature description
Enhance DLT's Redshift destination to automatically propagate column descriptions defined within the DLT column schema to the Redshift system table pg_description. This feature is essential for maintaining consistency between DLT's schema definition and the live Redshift catalog, significantly improving data discoverability and documentation.
### Are you a dlt user?
Yes, I'm already a dlt user.
### Use case
Data governance and documentation rely on clear and accurate column descriptions in Redshift.
Currently, DLT lacks a mechanism to push schema-defined descriptions to Redshift during data loading. This forces data teams to either manually update column descriptions or implement custom workarounds, which introduces a high risk of inconsistency and increases operational overhead across multiple tables.
### Proposed solution
Execute `COMMENT ON COLUMN` DDL upon schema update. Implement a mechanism in the Redshift destination that explicitly checks for the presence of a description field within the DLT column schema. If a description is present, DLT should execute the corresponding Redshift DDL command (`COMMENT ON COLUMN table.column IS 'description'`) immediately after the table or column is created or updated. This process should be executed as part of the schema evolution transaction to ensure descriptions are applied consistently and atomically with the structural changes.
### Related issues
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.