Add column documentation support for Iceberg destination tables
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### Feature description
Add support for column documentation/comments in Iceberg destination tables. Currently, when using the filesystem destination with `table_format="iceberg"`, there's no mechanism to propagate column descriptions from the dlt schema to the Iceberg table metadata.
This feature would enable automatic column comment propagation, similar to what's already implemented for the Databricks destination.
### Are you a dlt user?
Yes, I'm already a dlt user.
### Use case
Column documentation is essential for:
- Data discoverability in data catalogs and BI tools
- Self-documenting data pipelines (schema as single source of truth)
- Compliance and data governance (documenting PII, sensitive fields)
Without this feature, teams must manually update column descriptions in their Iceberg catalog, leading to inconsistency and operational overhead.
### Proposed solution
Propagate column descriptions from dlt schema to Iceberg using PyIceberg's schema APIs:
1. Extract `description` field from dlt column schema
2. Add to PyArrow schema via field metadata (`b"doc"` key)
3. PyIceberg's `pyarrow_to_schema()` auto-converts to `NestedField.doc`
4. For existing tables: use `update_column(col_name, doc=doc)`
**Example:**
```python
@dlt.resource(columns={
"order_id": {"description": "Unique sales order identifier"},
"customer_id": {"description": "Customer identifier"},
"order_date": {"description": "Order date"},
})
def orders():
yield {"order_id": 1, "customer_id": 100, "order_date": "2024-01-01"}
iceberg_adapter(orders, table_properties={"comment": "Sales orders"})
pipeline.run(orders)
```
### Related issues
_No response_
Contributor guide
Research direction
Start at the filesystem destination's Iceberg handling and compare it with the existing Databricks column documentation implementation. Trace how dlt column descriptions reach the PyArrow schema and how PyIceberg's pyarrow_to_schema() and update_column() expose documentation. Done means descriptions are present in new and existing Iceberg table metadata and the behavior is covered by the destination's tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100