Tokenizer (hasCreate): support delimiters + paste of arbitrary delimited text
- Dominant language
- TypeScript
- Stars
- 13k
- Forks
- 1.1k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 690
Description
## Summary
For a Tokenizer with `hasCreate` (free-text token creation), add support for **delimiters** (split typed/pasted text into multiple tokens on characters like comma, space, newline, semicolon) and **paste handling** (pasting a blob of delimited text creates one token per value). Common for email/tag inputs where users paste a list.
## Current state
`hasCreate` already enables free-text creation, but only one token at a time via Enter:
- `hasCreate?: boolean` — "When true, pressing Enter with text in the input commits the typed value" (`Tokenizer.tsx`). It appends a synthetic "Create: X" item and commits on Enter.
- **No delimiter handling** — typing `a, b, c` and hitting Enter creates a single token `"a, b, c"`, not three.
- **No paste handling** — there's no `onPaste` on the input, so pasting `a@x.com, b@y.com, c@z.com` drops the whole string into one token (or the query) instead of splitting into three.
## Desired behavior (with `hasCreate` on)
- **Delimiters:** as the user types, hitting a configured delimiter (default set: `,` and newline; consider space/`;` as opt-in) commits the current text as a token and starts a fresh one — like Enter, but mid-typing.
- **Paste:** pasting delimited text splits on the delimiters and creates one token per non-empty, trimmed value. Empty segments and dupes are dropped (respecting the existing `isAtMax`/duplicate guards already in the create path).
- **Trimming & de-dupe:** reuse the existing create-path guards (`query.trim()`, `selectedIds` duplicate check) so pasted values follow the same rules as typed ones.
- **Fallback:** if `hasCreate` is off, paste behaves as today (no token creation) — delimiters/paste-split only apply in create mode.
## Proposed API
New prop to configure the split characters, e.g.:
```tsx
```
- Default `delimiters` to a sensible set (comma + newline) so paste-a-list "just works" when `hasCreate` is on.
- Accept either a list of delimiter strings or a RegExp for advanced splitting.
## Prop-shape question — `hasCreate` in this context
Delimiter + paste-split behavior is meaningful **only** when free-text creation is allowed, so it naturally lives under `hasCreate`. For now: **reuse `hasCreate`** — delimiters/paste-split are active whenever `hasCreate` is true, gated further by the presence of a `delimiters` prop.
But it's worth reconsidering the prop as this grows: `hasCreate` is a boolean, and we're now layering multiple free-text behaviors (create-on-Enter, create-on-delimiter, create-on-paste). If this keeps expanding, a richer shape (e.g. `create={{ onEnter, delimiters, onPaste }}` or a `freeText` config object) may read better than stacking booleans. **Not blocking** — reuse `hasCreate` + add `delimiters` now; flag the shape as a spec follow-up if more create-related knobs appear.
## Acceptance criteria
- [ ] With `hasCreate`, typing a delimiter char commits the current text as a token and continues a fresh input.
- [ ] With `hasCreate`, pasting delimited text (e.g. `a@x.com, b@y.com`) creates one token per value.
- [ ] Pasted values are trimmed; empty segments and duplicates are dropped; `isAtMax` is respected.
- [ ] `delimiters` is configurable (default comma + newline); accepts strings and/or RegExp.
- [ ] With `hasCreate` off, paste/typing behavior is unchanged.
- [ ] Enter-to-create still works alongside delimiter-to-create.
- [ ] A11y: token additions announced consistently with the existing create flow; no double-commit of the final segment.
## Notes
- New public prop (`delimiters`) + behavior → spec + document (ties into the contract-testing work in #4163; note the input family already has an undocumented `startIcon`/`width` gap there).
- Sibling: `MultiSelector`/`PowerSearch` share tokenizer-like input surfaces — decide whether delimiter/paste support should be shared or Tokenizer-only.
Contributor guide
Assessment
This issue has not been assessed yet.