facebook / facebook/astryx

Tokenizer (hasCreate): support delimiters + paste of arbitrary delimited text

Open
#4,175 1 comment 0 reactions 1 assignee Claimed by @cixzhang View on GitHub
Dominant language
TypeScript
Stars
13k
Forks
1.1k
Avg merge
1d 15h
Merged PRs (30d)
690

Description

## Summary

For a Tokenizer with `hasCreate` (free-text token creation), add support for **delimiters** (split typed/pasted text into multiple tokens on characters like comma, space, newline, semicolon) and **paste handling** (pasting a blob of delimited text creates one token per value). Common for email/tag inputs where users paste a list.

## Current state

`hasCreate` already enables free-text creation, but only one token at a time via Enter:

- `hasCreate?: boolean` — "When true, pressing Enter with text in the input commits the typed value" (`Tokenizer.tsx`). It appends a synthetic "Create: X" item and commits on Enter.
- **No delimiter handling** — typing `a, b, c` and hitting Enter creates a single token `"a, b, c"`, not three.
- **No paste handling** — there's no `onPaste` on the input, so pasting `a@x.com, b@y.com, c@z.com` drops the whole string into one token (or the query) instead of splitting into three.

## Desired behavior (with `hasCreate` on)

- **Delimiters:** as the user types, hitting a configured delimiter (default set: `,` and newline; consider space/`;` as opt-in) commits the current text as a token and starts a fresh one — like Enter, but mid-typing.
- **Paste:** pasting delimited text splits on the delimiters and creates one token per non-empty, trimmed value. Empty segments and dupes are dropped (respecting the existing `isAtMax`/duplicate guards already in the create path).
- **Trimming & de-dupe:** reuse the existing create-path guards (`query.trim()`, `selectedIds` duplicate check) so pasted values follow the same rules as typed ones.
- **Fallback:** if `hasCreate` is off, paste behaves as today (no token creation) — delimiters/paste-split only apply in create mode.

## Proposed API

New prop to configure the split characters, e.g.:

```tsx

```

- Default `delimiters` to a sensible set (comma + newline) so paste-a-list "just works" when `hasCreate` is on.
- Accept either a list of delimiter strings or a RegExp for advanced splitting.

## Prop-shape question — `hasCreate` in this context

Delimiter + paste-split behavior is meaningful **only** when free-text creation is allowed, so it naturally lives under `hasCreate`. For now: **reuse `hasCreate`** — delimiters/paste-split are active whenever `hasCreate` is true, gated further by the presence of a `delimiters` prop.

But it's worth reconsidering the prop as this grows: `hasCreate` is a boolean, and we're now layering multiple free-text behaviors (create-on-Enter, create-on-delimiter, create-on-paste). If this keeps expanding, a richer shape (e.g. `create={{ onEnter, delimiters, onPaste }}` or a `freeText` config object) may read better than stacking booleans. **Not blocking** — reuse `hasCreate` + add `delimiters` now; flag the shape as a spec follow-up if more create-related knobs appear.

## Acceptance criteria

- [ ] With `hasCreate`, typing a delimiter char commits the current text as a token and continues a fresh input.
- [ ] With `hasCreate`, pasting delimited text (e.g. `a@x.com, b@y.com`) creates one token per value.
- [ ] Pasted values are trimmed; empty segments and duplicates are dropped; `isAtMax` is respected.
- [ ] `delimiters` is configurable (default comma + newline); accepts strings and/or RegExp.
- [ ] With `hasCreate` off, paste/typing behavior is unchanged.
- [ ] Enter-to-create still works alongside delimiter-to-create.
- [ ] A11y: token additions announced consistently with the existing create flow; no double-commit of the final segment.

## Notes
- New public prop (`delimiters`) + behavior → spec + document (ties into the contract-testing work in #4163; note the input family already has an undocumented `startIcon`/`width` gap there).
- Sibling: `MultiSelector`/`PowerSearch` share tokenizer-like input surfaces — decide whether delimiter/paste support should be shared or Tokenizer-only.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.