Make tokenization and stop-words pluggable for non-English tool catalogs and conversations

Summary

Make _utils.tokenize() and the STOPWORDS set configurable (per-language stop-word sets, optional CJK-aware tokenization fallback, language-neutral mode) so routing and dedup quality does not silently degrade on non-English catalogs.

Why this matters

Tool descriptions, user queries, and conversation histories in Portuguese, Spanish, German, Japanese, or Chinese hit an English-only stop-word list and a whitespace/word-regex tokenizer. For CJK text (no spaces) TF-IDF/BM25 tokenization collapses, making routing effectively random; for European languages, untreated stop-words skew scores. The library is marketed framework-agnostic and provider-agnostic — language is the remaining hidden assumption.

Current evidence

• src/contextweaver/_utils.py — module docstring declares it the single source of truth for tokenization; STOPWORDS is a frozen set of ~100 English words; tokenize() is regex word-splitting.
• context/dedup.py (Jaccard over tokenize() output) and all three scorers inherit the limitation.
• No open or closed issue mentions i18n/multilingual/language (checked all titles).

External context

BM25 implementations (e.g. rank-bm25) are tokenizer-agnostic; common practice is pluggable analyzers; CJK requires n-gram or dictionary segmentation. A stdlib-only char-bigram fallback for non-spaced scripts is a known lightweight approach.

Proposed implementation

1. Introduce a Tokenizer Protocol (in protocols.py) with the current behavior as DefaultEnglishTokenizer.
2. Thread an optional tokenizer through TfIdfScorer/BM25Scorer/FuzzyScorer constructors and dedup (default unchanged).
3. Add language="auto"|"en"|"neutral" convenience: neutral skips stop-words; auto detects non-spaced scripts via Unicode ranges and applies char-bigram tokenization (stdlib only — no new deps, consistent with the minimal-core-deps invariant).
4. Add a small multilingual fixture catalog and routing tests; optionally a scorecard scenario.

AI-agent execution notes

• Inspect first: _utils.py, context/dedup.py, routing/router.py _ensure_index, protocols.py, tests/test_utils.py, tests/test_dedup.py.
• Preserve: deterministic outputs; English defaults byte-identical; do not duplicate similarity logic outside _utils.py (AGENTS.md strong pattern).
• Failure modes: mixed-language catalogs (apply per-document detection consistently); stop-word sets must be versioned data, not network-fetched.

Acceptance criteria

• Custom tokenizer injectable into Router scoring and dedup with documented API.
• CJK fixture test: routing a Japanese query over a Japanese catalog returns the gold tool in top-5 (vs. failing baseline test recorded first).
• English default path byte-identical (full regression suite green).
• No new core dependencies.

Test plan

Unit tests per script family (Latin-with-stopwords, CJK); dedup behavior test; determinism test; benchmark informational run.

Documentation plan

docs/troubleshooting.md (non-English catalogs), docs/tool_router.md, CHANGELOG, AGENTS.md _utils.py row update.

Migration and compatibility notes

Not expected to require migration (defaults unchanged).

Risks and tradeoffs

Half-measures risk overpromising multilingual quality — document clearly that this is lexical-level support and embeddings ([embeddings] extra) remain the high-quality multilingual path; maintenance of stop-word data (keep to a few curated sets).

Suggested labels

product, ecosystem, good-first-ai-issue

Start with src/contextweaver/_utils.py, protocols.py, context/dedup.py, routing/router.py, and the listed tests to map tokenization and scorer flow. Establish the current English behavior first, then use the proposed multilingual fixtures and regression tests to verify injectable tokenizers, CJK routing, deterministic deduplication, unchanged English defaults, and documentation updates.