Form-framework evaluation for Astryx: which library should be the default?

# Form-framework evaluation for Astryx: which library should be the default?

> Results from a controlled vibe-test comparing four form frameworks on a constant Astryx design system. Preserved here (rather than in the repo) for easy reading and comment. The evaluation harness, the `@astryxdesign/formentor` concept scaffold, the three parity adapters, and the interactive playground live on branch `feat/formentor-scaffold` for reproduction only — they are **not** intended to merge.

## Abstract

Astryx ships a mature component library but takes no position on form state management and validation. This study asks which form framework should be the blessed default for Astryx. We designed a controlled experiment that holds the design system constant — every candidate renders the identical Astryx input components — and varies only the form-framework layer wired to those components. Four candidates were evaluated: **Formentor** (a schema-aware form layer purpose-built for Astryx), **Formisch**, **TanStack Form**, and **React Hook Form (RHF)**. To measure how readily each framework can be picked up in practice, all form code was generated by blank-slate LLM agents that learned each API cold from its package README, with no prior context. Scoring combined a deterministic TypeScript compile check with five form-specific quality dimensions, evaluated blind to framework identity. Across 15 realistic prompts per framework (60 generated forms), the candidates clustered tightly on correctness (87–93% clean compilation) and near-tied on form quality (98–100 / 100). The discriminating signals were code conciseness and cognitive load, where Formentor produced the shortest solutions (median 49 lines vs. 75 for TanStack) and required the fewest documentation lookups. **We recommend adopting Formentor as the Astryx default, contingent on two fixes identified by the study.**

## 1. Background

Forms are among the most common and most error-prone surfaces in tools built on a design system. Astryx provides high-quality input primitives (TextInput, Selector, NumberInput, CheckboxInput, TextArea, and a Field wrapper) but deliberately leaves form orchestration — state, validation, submission, and cross-field logic — to the consumer. Consumers therefore reach for a general-purpose form library and hand-wire it to Astryx components. This creates an unmanaged seam: every team re-solves the same schema-to-UI wiring, inconsistently.

Formentor is built on a distinctive premise: **schema awareness**. Rather than managing form state as its primary job, it treats a runtime schema as a reified type and crosses it with a design system's components via an InputSet, deriving renderable, validated fields automatically. The question for Astryx is whether this schema-aware model delivers enough value over mature general-purpose libraries to justify adopting it as the default.

## 2. The four candidates

The four candidates occupy meaningfully different points in the design space. Understanding these differences is prerequisite to a fair comparison, because each expresses the same concept (a bound, validated field) through a different idiom.

- **Formentor** — core thesis: schemas are types available at runtime (reified). A schema is defined once and crossed with a design system's InputSet, which maps each schema type to a component and supplies common props and field-wrapper chrome. Validation follows a ladder: schema constraints, field-level validators, cross-field (form-level) validators targeting nested fields, and submit-time async validation.
- **Formisch** — schema-first on Valibot; smallest bundle; one source of truth; multi-framework.
- **TanStack Form** — fine-grained validation timing and subscription-based state; multi-framework.
- **React Hook Form** — the most widely adopted of the four. Emphasizes uncontrolled/ref-based performance, uses a `Controller` to bridge controlled component libraries, and validates via a resolver (commonly a Zod resolver). Its maturity and community familiarity make it the pattern LLMs recognize most readily.

| | Formentor | Formisch | TanStack | RHF |
|---|---|---|---|---|
| Framework support | React | 6 frameworks | 6 frameworks | React |
| Distinctive | Schema↔component InputSets | Smallest bundle, one source of truth | Fine-grained validation timing | Maturity, community |

## 3. Method

**Design-system-constant.** Every target renders the identical Astryx components. Only the form-framework layer varies. This isolates the framework as the independent variable and removes component quality as a confound.

**Equal-effort adapter parity (Option B).** Formentor ships a native Astryx InputSet; the other three are design-system-agnostic and must be wired to Astryx. To compare inherent ergonomics fairly rather than merely rewarding Formentor for having an adapter, each non-Formentor framework received an equal-effort Astryx adapter: the same five bridge components (text, textarea, number, checkbox, select) binding to the same core inputs, each keeping its framework's native validation idiom. No framework was pre-integrated more than another, and none was hobbled.

**Blank-slate generation.** Each of the 60 forms (15 prompts × 4 frameworks) was generated by a freshly spawned agent with zero prior context: no conversation history, no knowledge of Formentor/Astryx, no memory of other prompts or targets. Its only inputs were the task prompt and the project it could explore; it learned the API from the package README exactly as a real consumer would. One agent per (prompt × framework); none reused.

**Blind, deterministic scoring.** Evaluation combines a TypeScript compile check (against a per-target config that resolves the real package sources) with five form-specific dimensions scored by pure functions: schema fidelity, validation correctness, form-grade accessibility, state and submission, and idiomatic use. Judgment logic is identical across frameworks; framework identity is stripped from the code before any identity-sensitive step. An anti-bias calibration confirms equal-quality reference solutions tie at 100 across all four frameworks (zero rubric bias).

## 4. Results (headline)

| Framework | Clean tsc compile | Form-quality score | Median LOC | Avg doc lookups |
|-----------|------------------:|-------------------:|-----------:|----------------:|
| **Formentor** | 87% (13/15) | 98 | **49** | **4.5** |
| **Formisch** | 93% (14/15) | 100 | 56 | 5.9 |
| **TanStack** | 93% (14/15) | 100 | 75 | 6.2 |
| **React Hook Form** | 87% (13/15) | 100 | 57 | 5.1 |

All four cluster within one prompt on compile rate and are near-tied on form-quality. The separating signals are **conciseness** and **cognitive load**, not correctness.

### A confound we caught and corrected

The first raw run showed TanStack/RHF at 60% compile — but every one of those failures was a **Zod v3-vs-v4 API mismatch** (the harness resolved a hoisted Zod v4 while the adapters install v3; agents wrote v3 syntax from training data), not a framework difference. Pointing the typecheck at the v3 copy the adapters actually resolve, and widening the RHF adapter's `control` type to accept transform resolvers, lifted both to a fair 87–93%. This is exactly the kind of asymmetry the fairness invariants exist to catch.

The residual failures are **genuine**:
- **Formentor (2):** one agent bug (an empty object passed to a string validator); one used `defaultValue` where the README documents `initialValue`.
- **RHF (2):** `z.literal(true)` for a required checkbox makes RHF infer the field as literally `true`, which then fights `defaultValues: false` — a real RHF+Zod sharp edge a naive author hits.
- **TanStack (1) / Formisch (1):** optional-field inference friction.

## 5. Qualitative findings (from reading all 60 solutions)

**Formentor is the terse, low-ceremony option.** Its schema-first model means a login form is 34 lines and `form.render()` handles layout, validation wiring, and the submit button. Agents needed the fewest doc lookups (4.5) and almost never hand-rolled state. For the common case (scalars, required/optional, format + length validation, cross-field, async submit) it was consistently the shortest correct answer, and its native `mode: 'view' | 'edit'` made the view/edit prompt trivial where others hand-rolled an `isEditing` toggle.

**TanStack is the most verbose.** `createFormHook` boilerplate + per-field render-props pushed it to a 75-line median. It is powerful and its subscription-based `canSubmit` gating is elegant, but it asks the most of a naive author.

**Formisch and RHF sit in between.** Both are schema-first (Valibot / Zod), both idiomatic, both familiar to the models. RHF's `Controller` + resolver is the most-recognized pattern; Formisch's Valibot-native cross-field checks handled dependencies cleanly.

**One real Formentor gap surfaced:** its `form.state.isValid` only reflects touched/submitted fields, so the "disable submit until valid" prompt forced a hand-rolled completeness gate — where RHF's `formState.isValid` and TanStack's `canSubmit` work out of the box. Worth fixing before Formentor is the default.

### Scaffold gaps the blank-slate agents found (a bonus)

Uncoached agents following the docs surfaced concrete, fixable gaps in the concept's own API surface:
- `ObjectSchemaImpl.getValuePath()` was documented in the README's cross-field example but did not exist — an agent followed the docs and hit a type error.
- Field-level `render({validate})` callbacks left validator params implicitly `any`; now typed.

These are exactly the kind of paper-cuts a concept scaffold hides until real, uncoached users exercise it — an argument for blank-slate evaluation as a design tool, not just a scoring tool.

## 6. Recommendation

**Adopt Formentor as the default, gated on two fixes.** On a constant Astryx design system, Formentor produced the most concise, lowest-cognitive-load forms for the common case and matched the field on correctness and form quality — while being the only candidate that unifies schema, validation, layout, and view/edit into a single artifact. The others are excellent general-purpose libraries but leave the schema↔form↔UI seam to the author.

Gate the default on:
1. **Fix `isValid` semantics** so "disable until valid" works without a hand-rolled gate (the one real ergonomic miss).
2. **Grow coverage.** This test only exercised standard scalar forms. Formentor must earn the advanced tier — field arrays, nested objects, conditional fields, wizards, typeahead, file upload — before it's the default for *all* forms. Until then, recommend Formentor for standard forms and keep an RHF or Formisch adapter as the escape hatch for advanced cases.

**If Formentor cannot absorb the advanced scope soon,** the strongest already-complete alternative is **Formisch**: schema-first like Formentor, tied for the best compile rate, terser than TanStack/RHF, and multi-framework.

## 7. Caveats

- Standard forms only — 15 prompts, scalar forms. Advanced patterns (field arrays, nested, wizards, typeahead, upload) were **not** run; the Formentor scaffold does not yet support them. Headline numbers do not speak to advanced forms.
- One generation per (prompt × framework); no repeat sampling for variance.
- Deterministic + tsc scoring only. A vision/design judge (rendered-output fidelity) was deferred and is not in these numbers.
- The Formentor here is a **concept scaffold**, not a production library; the server-side schema-derivation half (server schema → codegen) is out of scope.
- Self-referential bias risk: the same effort authored the scaffold, its README, the adapters, and the rubric. Mitigated by the anti-bias calibration (equal-quality solutions tie) and by holding evaluator judgment identical across targets, but not fully eliminable.

## Reproduction

All artifacts live on branch `feat/formentor-scaffold` (not for merge). To run the 4-way generation and the interactive playground:

```
node internal/vibe-tests/src/setup-forms.mjs --core-only
# spawn one blank-slate agent per generated task, then:
node internal/vibe-tests/src/build-playground.mjs
pnpm -F @astryxdesign/vibe-tests-app playground # http://localhost:5176/?mode=playground
```