Proposal: Collation support for string types
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
## Summary
Add column-level collation to the Iceberg spec: a `string` field can carry a provider-qualified collation (e.g. `icu.en_US-ci`) that defines case-insensitive, accent-insensitive, or locale-aware comparison and ordering. Storage stays UTF-8 — only comparison changes. Collation-aware min/max are stored so collated columns stay prunable.
## Motivation
Iceberg strings compare by UTF-8 bytes only — there is no way to declare a column case-/accent-insensitive or locale-sorted. Snowflake and Databricks users cite this as a migration blocker. Query-rewrite workarounds (`UPPER()`/`LOWER()`, `ILIKE`) don't scale and defeat min/max pruning.
## Design (current direction)
- A `collation` annotation on the string field, stored **unversioned** in the schema so any compatible engine can read the table.
- Collation-aware bounds are stored as ordinary `content_stats` under schema-declared collation metric fields (`collation-metrics`: `(collation, version) -> metric field id`); original values (not sort keys), tight, tagged with the collation implementation version they were selected under.
- Byte-order `lower_bounds`/`upper_bounds` are still written for collation-unaware engines. Readers prune with a collation metric field's bounds only on an exact collation + version match; otherwise they scan. A version mismatch degrades pruning, never correctness.
- Collation is comparison-only: partition/bucket transforms, equality-delete matching, and identifier-field equality stay binary. Providers are a registered set (like geo), `icu` to start.
## Artifacts
- Spec PR: https://github.com/apache/iceberg/pull/16972
- Reference implementation (iceberg-go): https://github.com/apache/iceberg-go/pull/1318
- Java POC: https://github.com/laskoviymishka/iceberg/tree/prototype/collation-support
- Proposal write-up: https://github.com/laskoviymishka/iceberg/blob/collation-support-proposal/format/collation-support-proposal.md
- Original proposal doc: https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit
- Dev-list thread: https://lists.apache.org/thread/44todz4x460g8pb89y8rpozlnmo8vdhc
Contributor guide
Research direction
Start with the Spec PR and format/collation-support-proposal.md, then compare the iceberg-go reference implementation and Java POC linked in the issue. The work is done when the collation design and compatibility behavior are resolved and reflected in the relevant specification and implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100