apache / apache/iceberg

Proposal: Collation support for string types

Open
#17,620 0 comments 0 reactions 0 assignees View on GitHub
Iceberg V4 proposal
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

## Summary

Add column-level collation to the Iceberg spec: a `string` field can carry a provider-qualified collation (e.g. `icu.en_US-ci`) that defines case-insensitive, accent-insensitive, or locale-aware comparison and ordering. Storage stays UTF-8 — only comparison changes. Collation-aware min/max are stored so collated columns stay prunable.

## Motivation

Iceberg strings compare by UTF-8 bytes only — there is no way to declare a column case-/accent-insensitive or locale-sorted. Snowflake and Databricks users cite this as a migration blocker. Query-rewrite workarounds (`UPPER()`/`LOWER()`, `ILIKE`) don't scale and defeat min/max pruning.

## Design (current direction)

- A `collation` annotation on the string field, stored **unversioned** in the schema so any compatible engine can read the table.
- Collation-aware bounds are stored as ordinary `content_stats` under schema-declared collation metric fields (`collation-metrics`: `(collation, version) -> metric field id`); original values (not sort keys), tight, tagged with the collation implementation version they were selected under.
- Byte-order `lower_bounds`/`upper_bounds` are still written for collation-unaware engines. Readers prune with a collation metric field's bounds only on an exact collation + version match; otherwise they scan. A version mismatch degrades pruning, never correctness.
- Collation is comparison-only: partition/bucket transforms, equality-delete matching, and identifier-field equality stay binary. Providers are a registered set (like geo), `icu` to start.

## Artifacts

- Spec PR: https://github.com/apache/iceberg/pull/16972
- Reference implementation (iceberg-go): https://github.com/apache/iceberg-go/pull/1318
- Java POC: https://github.com/laskoviymishka/iceberg/tree/prototype/collation-support
- Proposal write-up: https://github.com/laskoviymishka/iceberg/blob/collation-support-proposal/format/collation-support-proposal.md
- Original proposal doc: https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit
- Dev-list thread: https://lists.apache.org/thread/44todz4x460g8pb89y8rpozlnmo8vdhc

Contributor guide

Open the contributing guide

Research direction

Start with the Spec PR and format/collation-support-proposal.md, then compare the iceberg-go reference implementation and Java POC linked in the issue. The work is done when the collation design and compatibility behavior are resolved and reflected in the relevant specification and implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.