dlt-hub / dlt-hub/dlt

Propagate Column Descriptions from DLT Schema to Redshift

Open
#3,545 0 comments 0 reactions 0 assignees View on GitHub
destination enhancement
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 15h
Merged PRs (30d)
38

Description

### Feature description

Enhance DLT's Redshift destination to automatically propagate column descriptions defined within the DLT column schema to the Redshift system table pg_description. This feature is essential for maintaining consistency between DLT's schema definition and the live Redshift catalog, significantly improving data discoverability and documentation.

### Are you a dlt user?

Yes, I'm already a dlt user.

### Use case

Data governance and documentation rely on clear and accurate column descriptions in Redshift.
Currently, DLT lacks a mechanism to push schema-defined descriptions to Redshift during data loading. This forces data teams to either manually update column descriptions or implement custom workarounds, which introduces a high risk of inconsistency and increases operational overhead across multiple tables.

### Proposed solution

Execute `COMMENT ON COLUMN` DDL upon schema update. Implement a mechanism in the Redshift destination that explicitly checks for the presence of a description field within the DLT column schema. If a description is present, DLT should execute the corresponding Redshift DDL command (`COMMENT ON COLUMN table.column IS 'description'`) immediately after the table or column is created or updated. This process should be executed as part of the schema evolution transaction to ensure descriptions are applied consistently and atomically with the structural changes.

### Related issues

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.