Discussion: Expanding Unicode Text Processing

The current roc-lang/unicode package provides low-level CodePoint operations and grapheme-cluster splitting. I would like to discuss expanding it with commonly needed Unicode functionality, including normalization, case mapping, text segmentation, character properties, and eventually locale-sensitive collation.

The APIs below are illustrative rather than final. These features could be implemented incrementally, and some already have focused issues of their own.

1. Normalization

Implement the four normalization forms from UAX 15:

Form : [NFC, NFD, NFKC, NFKD]
QuickCheck : [Yes, No, Maybe]

normalize : Str, Form -> Str
quick_check : Str, Form -> QuickCheck
is_normalized : Str, Form -> Bool

canonically_equivalent : Str, Str -> Bool

An identifier-oriented convenience may also be useful:

nfkc_case_fold : Str -> Str

Compatibility equivalence could be exposed separately if there is a clear use case, since compatibility normalization intentionally removes some distinctions.

2. Case Mapping and Folding

String casing functions should support full mappings, including cases where one scalar maps to multiple scalars:

to_upper : Str, Locale -> Str
to_lower : Str, Locale -> Str
to_title : Str, Locale -> Str

Case folding should distinguish the default and Turkic behaviours:

FoldMode : [Default, Turkic]

case_fold : Str, FoldMode -> Str
canonically_case_insensitive_equal : Str, Str -> Bool

The equality operation should implement canonical caseless matching rather than simply comparing two unnormalized folded strings.

The exact titlecasing behaviour should document which word-boundary algorithm and locale tailoring it uses.

3. Text Segmentation

UAX 29 defines grapheme, word, and sentence boundaries. Since these algorithms fundamentally produce boundaries, the primary APIs could expose UTF-8 byte offsets or half-open byte ranges:

Range : {
    start : U64,
    end : U64,
}

grapheme_ranges : Str -> List(Range)
word_boundaries : Str -> List(U64)
sentence_boundaries : Str -> List(U64)

Fold-, callback-, cursor-, or iterator-like variants could let callers stop early and avoid allocating the complete result.

Convenience functions returning List(Str) could be layered on top. Functions such as words should clearly specify whether whitespace, punctuation, and other non-word segments are retained or filtered.

Line breaking

UAX 14 determines possible and mandatory line-break positions, rather than performing text layout itself:

LineBreakKind : [Allowed, Mandatory]

LineBreak : {
    byte_offset : U64,
    kind : LineBreakKind,
}

line_breaks : Str -> List(LineBreak)

Actual wrapping depends on available width, font measurement, and application-specific layout policy.

4. Scalar Properties

Typed foundational properties could be exposed alongside clearly defined convenience predicates:

general_category : Scalar -> GeneralCategory

is_alphabetic : Scalar -> Bool
is_white_space : Scalar -> Bool
is_decimal_digit : Scalar -> Bool

numeric_type : Scalar -> NumericType
numeric_value : Scalar -> [None, Value(Rational)]

script : Scalar -> Script
script_extensions : Scalar -> List(Script)

is_xid_start : Scalar -> Bool
is_xid_continue : Scalar -> Bool

Script should be generated from the selected Unicode version and include all defined scripts and special values. Script_Extensions is also important because some characters are associated with multiple scripts.

Emoji should expose the individual Unicode properties rather than a single ambiguous is_emoji predicate:

EmojiProperties : {
    emoji : Bool,
    emoji_presentation : Bool,
    emoji_modifier : Bool,
    emoji_modifier_base : Bool,
    emoji_component : Bool,
    extended_pictographic : Bool,
}

emoji_properties : Scalar -> EmojiProperties

Recognizing complete emoji sequences—such as flags, keycaps, modifier sequences, and RGI ZWJ sequences—would be a separate sequence-level API.

5. Locale Identifiers

Rather than enumerating languages in a tag union, locales could use an opaque, validated Unicode locale identifier based on BCP 47 and UTS 35:

Locale

root : Locale
parse : Str -> Result(Locale, InvalidLocale)
to_str : Locale -> Str

Examples:

german = Locale.parse("de")?
german_phonebook = Locale.parse("de-u-co-phonebk")?
turkish = Locale.parse("tr")?

This allows language, script, region, variants, and Unicode locale extensions to be represented without adding a new Roc tag for every locale or collation type.

6. Collation

Locale-sensitive collation requires the Unicode Collation Algorithm together with CLDR tailoring.

A reusable collator could avoid repeatedly processing the locale and options:

CollationStrength : [
    Primary,
    Secondary,
    Tertiary,
    Quaternary,
    Identical,
]

CaseFirst : [Default, UpperFirst, LowerFirst]

CollationOptions : {
    strength : CollationStrength,
    numeric : Bool,
    case_first : CaseFirst,
}

default_options : CollationOptions

Collator

collator : Locale, CollationOptions -> Result(Collator, CollationErr)

compare : Collator, Str, Str -> [Lt, Eq, Gt]
sort_key : Collator, Str -> List(U8)
sort : Collator, List(Str) -> List(Str)

Example:

names = ["Beere", "Ähre", "Apfel"]

german = Locale.parse("de")?
standard = Collation.collator(german, default_options)?
sorted_standard = Collation.sort(standard, names)

phonebook_locale = Locale.parse("de-u-co-phonebk")?
phonebook = Collation.collator(phonebook_locale, default_options)?
sorted_phonebook = Collation.sort(phonebook, names)

The API should specify:

• Whether sorting is stable when strings compare equal at the selected strength.
• Whether unsupported locale options produce an error or fall back.
• That collation equality is different from binary string equality.
• Whether sort keys may be persisted across package, Unicode, or CLDR upgrades.

Additional CLDR options could be added later.

7. Bidirectional Text

Correct display of mixed left-to-right and right-to-left text requires the Unicode Bidirectional Algorithm from UAX 9.

The API should distinguish between:

• Scalar bidirectional properties
• Paragraph-level analysis
• Per-line visual reordering

This could remain a separate focused proposal while sharing the same generated Unicode property tables.

8. Identifier and Security Support

Properties such as XID_Start, XID_Continue, Pattern_Syntax, and Pattern_White_Space would be useful for parsers and developer tooling.

Higher-level security operations from UTS 39—such as confusable skeletons and mixed-script detection—may belong in a separate Unicode.Security module or package, so they are not confused with ordinary string comparison.

Cross-Cutting Considerations

Unicode scalar values

Unicode algorithms such as normalization, casing, properties, and segmentation operate on Unicode scalar values rather than surrogate code points.

It may therefore be useful to expose a validated type:

Scalar

from_code_point : CodePoint -> Result(Scalar, [Surrogate])
to_code_point : Scalar -> CodePoint

Alternatively, each API must clearly document how surrogate code points are handled.

Versioning and conformance

The package should:

• Pin a specific Unicode version for its algorithms and data.
• Report the supported Unicode, Emoji, and, where applicable, CLDR versions.
• Avoid combining data files from different releases.
• Run the official Unicode conformance tests for each implemented algorithm.

Allocation-conscious APIs

Where practical, APIs should support scanning, folding, or returning ranges into the original UTF-8 string rather than requiring copied substrings or complete allocated lists.

Convenience allocation-heavy APIs can then be implemented on top.

Default behaviour and tailoring

The package should distinguish between:

• Default Unicode behaviour
• Locale-tailored behaviour
• Application-specific custom behaviour

Locale tailoring should not silently change an API documented as implementing the default Unicode algorithm.

Unicode data and CLDR data

Normalization, scalar properties, and default segmentation rely mainly on Unicode Character Database data.

Locale-sensitive casing and collation also require CLDR data, which may add significant package size. It may therefore make sense to place CLDR-backed functionality in a separate package or optional data modules.

Possible Implementation Phases

1. Unicode version manifest, generated-data pipeline, and official conformance tests.
2. Range-based grapheme segmentation and foundational scalar properties.
3. Normalization and default case mapping and folding.
4. Word and sentence boundaries, line breaking, and bidirectional processing.
5. Locale representation and a limited set of CLDR-backed casing and collation capabilities.
6. Additional locale tailorings, identifier utilities, and optional security functionality.

Questions for the Team

1. Should the package expose a distinct Scalar type?
2. Should range- or fold-based APIs be the primary segmentation interface, with List(Str) functions as conveniences?
3. Should CLDR-backed functionality live in roc-lang/unicode or a separate package?
4. Is supporting all CLDR locales realistic, or should locale data be modular?
5. Which subset would provide the best first contribution: normalization, casing, segmentation, or foundational properties?

The issue names no files, tests, or entry points; begin by narrowing the proposal to one phase and locating the package’s existing CodePoint and grapheme-cluster APIs. Define the chosen Unicode version and conformance tests before implementation; done means a focused API, documented behavior, and passing official tests.