dlt-hub / dlt-hub/dlt

Add column documentation support for Iceberg destination tables

Open
#3,791 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### Feature description

Add support for column documentation/comments in Iceberg destination tables. Currently, when using the filesystem destination with `table_format="iceberg"`, there's no mechanism to propagate column descriptions from the dlt schema to the Iceberg table metadata.

This feature would enable automatic column comment propagation, similar to what's already implemented for the Databricks destination.

### Are you a dlt user?

Yes, I'm already a dlt user.

### Use case

Column documentation is essential for:
- Data discoverability in data catalogs and BI tools
- Self-documenting data pipelines (schema as single source of truth)
- Compliance and data governance (documenting PII, sensitive fields)

Without this feature, teams must manually update column descriptions in their Iceberg catalog, leading to inconsistency and operational overhead.

### Proposed solution

Propagate column descriptions from dlt schema to Iceberg using PyIceberg's schema APIs:

1. Extract `description` field from dlt column schema
2. Add to PyArrow schema via field metadata (`b"doc"` key)
3. PyIceberg's `pyarrow_to_schema()` auto-converts to `NestedField.doc`
4. For existing tables: use `update_column(col_name, doc=doc)`

**Example:**
```python
@dlt.resource(columns={
"order_id": {"description": "Unique sales order identifier"},
"customer_id": {"description": "Customer identifier"},
"order_date": {"description": "Order date"},
})
def orders():
yield {"order_id": 1, "customer_id": 100, "order_date": "2024-01-01"}

iceberg_adapter(orders, table_properties={"comment": "Sales orders"})
pipeline.run(orders)
```

### Related issues

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at the filesystem destination's Iceberg handling and compare it with the existing Databricks column documentation implementation. Trace how dlt column descriptions reach the PyArrow schema and how PyIceberg's pyarrow_to_schema() and update_column() expose documentation. Done means descriptions are present in new and existing Iceberg table metadata and the behavior is covered by the destination's tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.