Define a panic-free, resource-bounded contract for public text and byte APIs

Summary

Establish one repository-wide contract for every public operation that processes caller-controlled Str, UTF-8 bytes, code points, ranges, or injected analysis facts:

• valid input never reaches crash, an “unreachable” catch-all, stack overflow, unchecked arithmetic, or silent partial output;
• malformed or unsupported input produces a precise typed result;
• work, stack, allocation, copying, and source-retention behavior are documented and testable;
• callers handling untrusted input have deterministic limits that fail before unsafe resource growth;
• convenience APIs and bounded/streaming APIs share the same implementation and semantics.

This is a cross-cutting safety and API contract, not a replacement for feature-specific correctness work. Grapheme conformance and known regressions remain in #19, #22, and #35; allocation-conscious grapheme ranges remain in #37; line-breaking rules remain in #38. Those implementations should satisfy the common contract defined here.

“Panic-free” does not mean hiding defects or claiming that a process can recover from an operating-system allocator abort. It means caller-controlled data is never used as the reason to panic, and checkable resource exhaustion is represented before it becomes integer overflow, runaway traversal, or avoidable out-of-memory termination.

Why this needs an explicit contract

The current public package exports CodePoint and Grapheme. Several implementation details show why conformance tests alone are not enough:

• Grapheme.split_help ends with a catch-all crash; #19 proves valid Unicode text can reach it.
• Grapheme.to_list_str crashes if reconstructed code points fail to encode, while #22 proves the state machine can already lose an input code point. Internal assumptions need structural enforcement plus public losslessness checks.
• CodePoint.internal_from_u32_unchecked is included in the exposed CodePoint module record. Its name says “internal”, but a caller can supply any U32; values above U+10FFFF reach its crash.
• CodePoint.append_utf8 accepts any CodePoint, including surrogate code points. Surrogates are code points but not Unicode scalar values and have no valid UTF-8 encoding. The function can therefore return ill-formed bytes despite being documented as UTF-8 encoding.
• CodePoint.utf8_len similarly reports a byte length for surrogate code points, while its declared InvalidCodePoint error is unreachable for values constructed through CodePoint.from_u32.
• The three-byte decoder checks answer < 0x80 for overlong input; a three-byte sequence must encode at least U+0800. Sequences such as a three-byte encoding of U+0080 therefore appear able to bypass the overlong check.
• parse_utf8_help, cps_to_str_help, and the grapheme state machine traverse caller-sized input recursively and repeatedly create/drop/append lists. Stack behavior and optimized versus unoptimized behavior are not documented.
• CodePoint.parse_utf8 preallocates a code-point list using the byte length. Grapheme.split then builds several additional full-input lists and reconstructed strings. The asymptotic and peak-allocation contract is currently implicit.
• Scalar.roc is not exported today, but contains TODO functions implemented with crash. A repository-wide audit should ensure unfinished crash stubs cannot accidentally become public later.

Roc's built-in Str.from_utf8 already demonstrates a useful diagnostic shape: malformed bytes produce a typed problem and byte index. The package should provide at least equivalent precision whenever it accepts raw bytes.

Failure taxonomy

Do not use one catch-all error for unrelated situations. Each public operation should be total or return a small operation-specific error type whose variants follow these categories:

Condition	Required public behavior	Examples
Valid Unicode text	Complete deterministic result; no UTF-8 error and no panic	Any Roc Str, including NUL, controls, noncharacters, private-use characters, long combining/ZWJ runs
Invalid byte input	Typed decode error at a deterministic byte offset	Invalid leading byte, stray/missing continuation, overlong form, surrogate encoding, value above U+10FFFF
Unsupported optional behavior	Typed Unsupported/MissingAnalyzer result when that behavior was explicitly requested	Requested language analyzer, tailoring, or data profile is unavailable
Checkable work/size exhaustion	Typed LimitExceeded before overflow or excessive allocation; never a result presented as complete	Input bytes, decoded scalars, output items, retained bytes, work units
Programmer misuse	Make invalid states unrepresentable where possible; otherwise return InvalidArgument	Reversed/out-of-bounds range, byte offset inside a scalar, unsorted or overlapping injected facts
Impossible internal invariant	Eliminate by types/exhaustive state transitions. If still detectible at runtime, return a stable internal-fault code at the public boundary rather than crashing from caller data	Corrupt generated table, impossible sealed-state transition

A normal end condition is not malformed input. For example, an empty byte slice passed to a “decode next” operation is better represented as End than Err(ListWasEmpty), unless the operation specifically promises that an item must be present.

An optional feature that was not requested should use its documented default. If a caller explicitly requests an analyzer/profile and it is unavailable, silently falling back is not acceptable.

Internal generator and test failures may stop generation loudly because they do not process runtime caller text. Production modules, however, must not depend on a crash path whose reachability varies with public input.

API and type guidance

Valid Str operations should be total where possible

A Roc Str is already valid UTF-8. Operations that only inspect a Str should not decode it through an API that invents a recoverable malformed-UTF-8 case. Prefer total folds/walkers and sealed internal scalar state.

An operation may still return a result when it accepts explicit resource limits, optional analyzer facts, or caller-supplied ranges. Its errors should describe those conditions, not claim the valid Str was malformed.

Raw byte operations need indexed decode errors

Unify decoding behavior around a stable diagnostic record conceptually like:

• offset : U64, identifying the first offending or incomplete byte;
• problem, with distinct tags for invalid start, unexpected end, expected continuation, overlong encoding, encoded surrogate, and code point above U+10FFFF.

The exact names can align with Roc's current Str.Utf8Problem. Results must be identical in debug and optimized builds and across supported platforms. Do not include memory addresses, opaque state dumps, or platform-specific exception strings in the stable value.

A partial decoder must consume either exactly one complete scalar or zero bytes on error/end, and report the same error offset as the full decoder at that position.

Encode Unicode scalar values, not arbitrary code points

UTF-8 encodes Unicode scalar values; surrogate code points are excluded. Use one of these type-safe designs:

• take a sealed Scalar value in infallible UTF-8 encoding functions; or
• keep accepting CodePoint, but return Err(NonScalar) for U+D800..U+DFFF.

The byte-count operation and the encoder must agree for every input. No public “unchecked” constructor should accept arbitrary caller values. If an unsafe constructor is needed between trusted modules, keep it outside the exported surface and prove its precondition at each call site.

Validate caller-supplied facts once

Range/profile/analyzer inputs introduced by #37, #38, or future APIs must be validated before use:

• checked start/end ordering and bounds;
• UTF-8 scalar alignment;
• sorting, uniqueness, and overlap policy;
• version/profile compatibility;
• no impossible enum or missing required metadata.

Validation failure is InvalidArgument, not a panic and not a silent normalization unless normalization is explicitly documented.

Resource limits and atomicity

Document unconditional algorithmic bounds for every public operation. In addition, provide bounded entry points for operations whose input or collected output can be attacker-controlled.

A common limits record may include:

• maximum input bytes;
• maximum decoded scalars;
• maximum output items/ranges;
• maximum deterministic work units;
• where relevant, maximum bytes copied or retained by the result.

Use operation-specific subsets rather than forcing irrelevant fields everywhere. Define exactly what one work unit means and at what point it is charged.

A limit error should include a stable resource kind, configured limit, and the offset/count where the operation stopped or the minimum required value when known. Check zero, one, exact-limit, and one-over-limit behavior.

Required rules:

• Use checked addition, multiplication, index advancement, capacity calculations, and byte/scalar offset conversion.
• Preflight known upper bounds before reserving or copying.
• For output whose size is discovered incrementally, check the limit before append/emission.
• A collecting API returns either a complete Ok result or an Err; it must not expose truncated output as success.
• A walking/streaming API must distinguish caller-requested early stop, clean completion, and limit exhaustion. If it exposes accumulated state on limit exhaustion, label that state explicitly as partial.
• Convenience and bounded APIs share one engine so limits cannot change Unicode semantics.
• Platform allocator failure may be unrecoverable in Roc. State this honestly. Bounded APIs are the prevention mechanism and must check all predictable growth before asking the allocator.

Traversal, stack, allocation, and retention contract

For every public operation, documentation and tests must state:

• worst-case time in input bytes/scalars and output size;
• maximum algorithmic stack depth;
• auxiliary allocation, excluding caller-requested output;
• number and size of full-input copies;
• whether returned strings/slices retain the caller's source allocation;
• whether early stop avoids decoding/classifying the suffix.

Caller-sized recursion is acceptable only if tail-call behavior is guaranteed for every supported build mode and verified with adversarial tests. Otherwise use iterative walks/folds or an explicit bounded stack. Recursive descent for future nested structures must have a caller-visible depth limit.

Avoid repeated drop_first, concatenation, rescanning, and append patterns unless uniqueness/ARC behavior proves linear work and bounded live memory. Deterministic work and allocation counters should be the primary CI evidence; optimized wall-clock/RSS measurements are supporting evidence.

Range outputs should refer to the source by integer offsets and should not retain it. Materialized substring APIs must document copies and seamless-slice retention. This complements rather than duplicates the zero-copy implementation work in #37.

Adversarial and failpoint tests

Add a shared safety suite used by every public text/byte API.

UTF-8 boundary matrix

Cover:

• every leading-byte category and representative continuation bytes;
• truncated two-, three-, and four-byte sequences at every byte position;
• stray continuation bytes;
• all overlong-length boundaries, including three-byte encodings of U+0080..U+07FF;
• encoded surrogates;
• U+10FFFF and values immediately above it;
• valid one-, two-, three-, and four-byte boundary scalars;
• concatenations where valid prefixes precede malformed suffixes.

Cross-check full and partial decoding against Roc's built-in UTF-8 validation. Assert exact error variant, byte offset, bytes consumed, and no accepted ill-formed sequence. Verify every scalar round-trips through encode/decode and every surrogate is rejected by UTF-8 encoders.

Long valid text

Run geometrically increasing inputs containing:

• ASCII and mixed-width UTF-8;
• one base followed by very long Extend/combining-mark runs;
• repeated ZWJ and extended-pictographic chains;
• very long RI and Prepend runs;
• long Hangul and Indic linker sequences;
• long spaces, controls, and property-transition patterns;
• nested paired punctuation, bidi embeddings/isolates, or recursive structures when public algorithms that use nesting are added.

Exercise empty, singleton, exact inline-storage boundaries, allocation growth boundaries, and large multi-allocation inputs. Run both debug and optimized code so correctness does not depend on optimization turning recursion into a loop.

Small budgets and synthetic overflow

For every limit kind, test budgets 0, 1, exact required, and exact-minus-one. Inject a failure at every allocation/emission/work checkpoint for a small representative input and assert:

• the same stable LimitExceeded category;
• no panic;
• no success containing a prefix presented as complete;
• no corrupted offsets or retained partially built object.

Test arithmetic near integer maxima using synthetic counters/helpers so enormous allocations are unnecessary. Where allocator failpoints are supported, fail each allocation in turn; otherwise the deterministic budget seam is the required failpoint mechanism.

Fuzzing and operational guards

Maintain separate fuzz targets for:

1. arbitrary byte lists through each byte-decoding entry point;
2. valid Unicode scalar sequences through every Str operation;
3. caller-supplied ranges/profiles/analyzer facts;
4. structured adversarial sequences for stateful Unicode rules.

Seed with official Unicode conformance data plus all minimized regressions from #19, #22, #35, #37, and #38. For each case assert:

• no panic, abort, stack overflow, hang, invalid index, or data loss;
• deterministic result/error and exact first-failure offset;
• successful transformations preserve all source data unless their contract explicitly transforms it;
• work, output, and allocation counters remain within the documented bound;
• bounded calls cannot exceed configured limits;
• encode/decode and range reconstruction round-trip invariants.

Run fuzz cases in a supervised process with wall-time and resident-memory caps so a regression becomes a reproducible test failure rather than taking down CI. Report the seed and code points/bytes, minimize failures, and persist the minimized corpus. A sanitizer/guarded-allocation build should be used when available.

Repository-wide audit checklist

Create and maintain a checked-in inventory covering each exported function and every helper reachable from it:

• input trust domain;
• total versus typed-result behavior;
• error variants and offsets;
• limits and atomicity;
• time/stack/allocation/copy/retention bounds;
• panic/crash reachability;
• deterministic tests and fuzz target.

Audit every crash, unchecked/wrapping arithmetic operation, unsafe constructor, recursive function, List.with_capacity/reserve, full-input conversion, slice, and silent error recovery. Classify each occurrence as:

• production public-path code to remove/replace;
• private invariant made unrepresentable and tested;
• build-time generator failure;
• test-only assertion;
• dead/incomplete code that must be removed or remain impossible to export.

CI should fail if a new exported operation lacks an inventory row or if a production module adds an unreviewed crash/unchecked public path.

Acceptance criteria

• The README defines the panic-free/resource-bounded contract and the failure taxonomy above without claiming recovery from unrecoverable host allocator failure.
• Every currently exported operation in CodePoint and Grapheme, plus new public APIs added by #37/#38, has a completed audit row for errors, limits, complexity, stack, allocation, copying, and retention.
• No catch-all crash, TODO panic, public unchecked constructor, or caller-sized unbounded recursion is reachable from any exported operation on valid input.
• CodePoint.internal_from_u32_unchecked is removed from the public surface or replaced with a safe typed constructor.
• UTF-8 encoding accepts only scalar values or returns NonScalar; append_utf8, byte-length calculation, and to_str agree for every code point.
• Full and partial UTF-8 decoding reject all malformed forms, including every overlong boundary, with a deterministic first-error byte offset and no consumed/returned partial scalar.
• Operations over valid Str are total unless they accept explicit limits or optional caller facts; they never return a spurious malformed-UTF-8 error.
• Unsupported requested behavior, invalid arguments, limit exhaustion, and detectable internal faults have distinct stable typed results; no silent fallback or partial-success result is used.
• All size/capacity/index arithmetic on public paths is checked before allocation, slicing, append, or offset advancement.
• Every allocating/collecting operation has documented linear-or-better bounds and either a bounded entry point or a documented hard maximum; streaming alternatives have bounded auxiliary storage and explicit completion/early-stop/limit statuses.
• Public traversal has a verified constant stack bound or an explicit tested depth limit in debug and optimized builds.
• Small-budget/failpoint tests exercise every check boundary and prove atomic error behavior.
• The adversarial UTF-8, long-sequence, recursion/stack, arithmetic-boundary, work-scaling, allocation, copying, and retention tests run in CI on all supported platforms.
• Reproducible byte, valid-text, and structured-input fuzzing runs on a schedule under time/RSS guards; minimized failures are retained.
• #19 and #22 remain focused regressions, while #35, #37, and #38 each demonstrate compliance with this shared contract without duplicating its infrastructure.
• CI enforces the public-API audit inventory and rejects unreviewed production crash paths or undocumented exported operations.

References

• Roc Str documentation and indexed UTF-8 errors
• Issue #19: valid grapheme input reaches a crash
• Issue #22: grapheme output loses a code point
• Issue #35: Unicode 17 grapheme conformance
• Issue #37: zero-copy grapheme byte ranges
• Issue #38: Unicode 17 line-break opportunities

Start by auditing the public entry points and named hotspots: Grapheme.split_help, Grapheme.to_list_str, CodePoint.parse_utf8, CodePoint.append_utf8, CodePoint.utf8_len, and Scalar.roc. Review the existing behavior against the failure taxonomy, resource-limit rules, and shared safety-suite requirements. Done means every public text/byte operation has documented bounds, typed failures, stable decoding behavior, and no caller-triggered crash or partial success.