Full-text search uses to_tsvector('simple'): diacritics and inflections silently return no results
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
`search_tsv` is generated with `to_tsvector('simple', content)`. The `simple` configuration does no
stemming and no accent folding, so message search misses obvious matches in any language where words
inflect or carry diacritics. On a Portuguese-speaking team this makes search unreliable for everyday
queries; English is affected too, just less visibly.
The migration comment says the config can be revisited *"behind evidence"* — this issue is that
evidence, measured on a self-hosted relay.
## Reproduction
Relay `ghcr.io/block/buzz:sha-00e61ea`, single-node `deploy/compose` stack, Postgres 17.
Post one message:
```
Manutenção programada nas estações de trabalho — configurações de rede serão revisadas
```
Then search via `buzz messages search --query `:
| Query | Expected | Actual |
|---|---|---|
| `manutenção` | match | ✅ match |
| `manutencao` (no diacritic) | match | ❌ **no results** |
| `estações` (as written) | match | ✅ match |
| `estação` (singular) | match | ❌ **no results** |
| `programada` | match | ✅ match |
Same behaviour through `POST /query` with a NIP-50 `search` filter.
## Why this matters in practice
- **Diacritics are typed inconsistently on mobile.** `manutencao` is how a large share of
Portuguese-speaking users will type it — including people who wrote the original message. Search
silently returns nothing, which reads as "the message is gone", not "your query was normalised
differently".
- **Inflection is not an edge case.** Singular/plural is the most common way a searcher's wording
differs from the author's. `estação` vs `estações` is not a clever query; it is the normal one.
- **English is affected as well**, just less noticeably: `run` does not match `running`,
`deploy` does not match `deployed`.
- The failure is **silent**. There is no "did you mean", no partial match, no hint that
normalisation is the reason — an empty result set is indistinguishable from "no such message".
## Root cause
`migrations/0001_initial_schema.sql`:
```sql
search_tsv TSVECTOR GENERATED ALWAYS AS (
CASE WHEN kind IN (1059, 30300, 30622, 44100, 44101) THEN NULL::tsvector
ELSE to_tsvector('simple', content)
END
) STORED,
```
with the accompanying comment:
> `'simple'` config = no stemming/stopwords, matching the existing substring-ish search semantics;
> the search lane can revisit the config behind evidence.
Query side matches it in `crates/buzz-search/src/query.rs`:
```rust
qb.push("websearch_to_tsquery('simple', ");
```
So index and query are consistent — the behaviour is deliberate, not a mismatch. The question is
whether `simple` is the right default once a deployment is single-language.
## Possible directions
Listing options rather than proposing a patch, since this is a product call and there are real
trade-offs:
1. **`unaccent` + `simple`** — fixes diacritics only, keeps language-neutral behaviour. Requires the
`unaccent` extension and an `IMMUTABLE` wrapper for the generated column. Smallest behaviour
change; does not address inflection.
2. **Operator-configurable text search configuration** (e.g. `BUZZ_FTS_CONFIG=portuguese`,
defaulting to `simple`). Self-hosted deployments are usually single-language, so the operator
knows the answer the product cannot. Changing it means rebuilding `search_tsv`.
3. **Keep `simple`, surface the constraint in the UI** — an empty result is currently
indistinguishable from "nothing matched". Even a hint that search is exact-form would remove most
of the confusion, at no schema cost.
Option 2 seems the best fit for self-hosted, but option 3 alone would already remove the "the
message disappeared" reading, which is the part that erodes trust in search.
## Environment
- Relay: `ghcr.io/block/buzz:sha-00e61ea`, NIP-11 reports `Buzz Relay 0.2.1`
- Deployment: `deploy/compose` single node, Postgres 17, self-hosted
- Client: `buzz-cli` built from the same commit
- Language: Brazilian Portuguese (pt-BR)
## Note on the privacy exclusions
Unrelated to the bug, but worth recording since I read the same code: the `CASE` excluding kinds
1059 / 30300 / 30622 / 44100 / 44101 from `search_tsv` means gift-wrapped DMs are not in the search
index at all (`NULL` tsvector never matches `@@`). That is a good property and it is easy to miss —
it deserves to stay whatever happens to the text search configuration.
Contributor guide
Assessment
This issue has not been assessed yet.