julia-script / julia-script/silk

stdlib: add grapheme traversal and locale-independent case mapping

Open
#42 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

new feature P2 spec-change stdlib
Dominant language
TypeScript
Stars
48
Forks
0
Avg merge
4h 49m
Merged PRs (30d)
213

Description

Context

silk/unicode now ships Unicode 17.0.0 NFC and NFD normalization. This issue tracks the
remaining post-stable Unicode functionality: extended grapheme-cluster traversal and full
locale-independent default upper/lower case mapping.

Locale-sensitive collation remains tracked separately in #162.

Current behavior

import silk.unicode { normalizeNfc, normalizeNfd }

Normalization is available and ordinary string equality remains exact. There is currently no
grapheme cursor and no upper- or lower-case transformation API.

Intended surface

pub fn nextGrapheme(
  value: string,
  cursor: GraphemeCursor
) -> Option<GraphemeStep>

pub effect fn toLowercase(
  value: string
) -> String ! OutOfMemoryError ? &mut Allocator

pub effect fn toUppercase(
  value: string
) -> String ! OutOfMemoryError ? &mut Allocator

GraphemeStep reports byte start/end offsets and the next cursor. It does not retain a borrowed
string view.

Requirements

  1. Grapheme traversal implements Unicode 17.0.0 extended grapheme clusters from UAX #29.
  2. Traversal is allocation-free and reports UTF-8 byte boundaries.
  3. Case conversion implements full locale-independent default lower/upper mappings, including
    contextual mappings.
  4. Case conversion allocates an owned String and exposes OutOfMemoryError plus Allocator.
  5. Language tailoring, case folding, title casing, and locale-sensitive collation are out of scope.
  6. Tables remain generated ordinary Silk data with no compiler privilege.
  7. Ordinary string equality remains exact and normalization-free.

Acceptance criteria

  • A base scalar and combining mark remain one grapheme.
  • Emoji ZWJ and regional-indicator conformance cases match Unicode 17.0.0 data.
  • Traversal returns exact UTF-8 byte offsets without allocating.
  • Lower/upper conversion covers a mapping that changes byte length.
  • Contextual default mappings are covered.
  • Evaluator, native LLVM, and direct Wasm agree.
  • Generation is deterministic and the compiler contains no Unicode-policy special case.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the silk/unicode implementation and the existing normalizeNfc and normalizeNfd surfaces. Review the proposed nextGrapheme, toLowercase, and toUppercase behavior against Unicode 17.0.0 and UAX #29, then verify agreement across the evaluator, native LLVM, and direct Wasm. Done means the listed grapheme, mapping, allocation, generation, and exact-equality criteria pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.