cockroachdb / cockroachdb/cockroach

Support CITEXT types with LIKE

Open
#149,791 3 comments 0 reactions 1 assignee Claimed by @BryanWangAtCRL View on GitHub
C-enhancement docs-known-limitation T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

CITEXT isn't currently supported when using the `LIKE` operator. This is because CITEXT is the case-insensitive collation, `"und-u-ks-level2"`, which is nondeterministic, [as determined here](https://github.com/cockroachdb/cockroach/blob/d847f6e6abc21f237c59297ac4d5a654a346e46c/pkg/util/collatedstring/collatedstring.go#L56-L64). Collated strings are only supported when they are deterministic, [as seen here](https://github.com/cockroachdb/cockroach/blob/d847f6e6abc21f237c59297ac4d5a654a346e46c/pkg/sql/sem/eval/match.go#L130-L136).

**Describe the solution you'd like**

Collated strings with the locale that is used for CITEXT, `"und-u-ks-level2"`, might be able to be canonicalized such that there exists a deterministic form of them, which would allow `LIKE` comparisons for CITEXT. Currently, I theorize that there are 2 sources of nondeterminism for *just this* specific locale:

1. The relation between uppercase & lowercase unicode characters, i.e: `a` and `A`, `è` and `È`.
2. The relation between unicode characters and their decomposed counterparts (accented e can be a single unicode character or 2 unicode code points, for example, è = \u00C8 but also è = \u0065\u0300).

If these are the only 2 sources of nondeterminism, then we can canonicalize CITEXT strings via a unicode-aware `lower()` function (which normalizes the nondeterminism introduced by 1) and perform unicode's NFD normalization function (which folds decomposed unicode code points into 1 unicode character, thus normalizing the nondeterminism introduced by 2). Assuming that this issue has correctly identified the sources of nondeterminism, then this canonicalized form of CITEXT would be deterministic and sufficient for use with the `LIKE` operator.

I do acknowledge that this is a quick 'hacky' solution until a locale-aware regex engine is introduced to Go, but would immediately benefit workflows which use CITEXT with LIKE expressions.

Jira issue: CRDB-52348

Epic CRDB-65892

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.