`/search` paginates in application memory, so every page costs the server everything up to that page

## Summary

`/search` fans out to two Atlas Search indexes as two separate `db.aggregate()` calls, then merges, deduplicates, sorts, and slices the combined result in Node (`controllers/search.js:282`). Each branch caps at `$limit: limit + skip` (`controllers/search.js:211,216`), so a request for page 40 makes Atlas produce and return 4000 documents per index and makes Node process up to 8000 of them in order to hand back 100.

`/query` pushes the same work down to Mongo — `db.find(props).limit(limit).skip(skip)` (`controllers/crud.js:87`) — and its latency is flat with depth. `/search` cannot, and its latency is not.

Nothing about `$search` requires this. A single Atlas Search pipeline sorts by score natively, returns each document once, and can end in `$skip`/`$limit`. The blocker is the two-index fan-out, not the search stage.

This is the performance half of the `/search` work. The correctness half is #307, which should land first and independently — a client can page `/search` honestly once that lands, just not cheaply.

## Why this matters

**Cost scales with depth, not page size.** Dev, `{"searchText":"line"}`, `limit=100`:

```text
skip=0 -> 100 documents in 379ms
skip=1000 -> 100 documents in 664ms
skip=2000 -> 100 documents in 781ms
skip=3000 -> 100 documents in 1172ms
skip=3700 -> 100 documents in 2021ms
skip=3800 -> 25 documents in 1954ms
skip=3900 -> 0 documents in 1784ms
skip=5000 -> 0 documents in 1802ms
skip=20000 -> 0 documents in 1815ms
```

`/query` at the same depths, `{"__rerum.APIversion":{"$exists":true}}`, `limit=100`:

```text
skip=0 -> 100 documents in 340ms
skip=1000 -> 100 documents in 296ms
skip=5000 -> 100 documents in 323ms
skip=20000 -> 100 documents in 369ms
skip=99000 -> 100 documents in 445ms
```

The two seconds spent to return *nothing* at `skip=5000` is the cost of producing and merging the entire matching set. Cost plateaus once the result set is exhausted, which confirms the bound is `min(total matches, limit + skip)` per branch. Production shows the same shape at smaller scale: `line` has 574 matches there, and the empty pages at `skip=1000`, `2000`, and `20000` still cost roughly 850-900 ms each.

**The ceiling is the `skip` maximum.** At the effective maximum of 100000, a sufficiently broad term at maximum depth asks Node to hold on the order of 100000 full RERUM documents per branch in memory and sort them. Not identifiers — whole documents.

**It is a shared-process cost.** This work happens in the API process, not in Atlas. Under PM2 cluster mode a few concurrent deep searches contend for the same workers that serve every other endpoint.

## Affected lines

| File | Line | Current |
|------|------|---------|
| `controllers/search.js` | 82 | `buildDualIndexQueries()` builds two independent pipelines |
| `controllers/search.js` | 84, 153 | Two separate indexes: `presi3AnnotationText`, `presi2AnnotationText` |
| `controllers/search.js` | 211, 216 | `$limit: limit + skip` per branch — the depth cost |
| `controllers/search.js` | 32 | `mergeSearchResults()` deduplicates in Node, after the per-branch cap |
| `controllers/search.js` | 282, 368 | `merged.slice(skip, skip + limit)` — pagination in application memory |
| `controllers/search.js` | 446, 540, 672 | Same shape in the unmounted `searchFuzzily`, `searchWildly`, `searchAlikes` |
| `controllers/crud.js` | 87 | `/query` for contrast: `.limit(limit).skip(skip)` in Mongo |

## Proposed change

Three options, best first. This issue should settle which one before any code is written.

### 1. One combined Atlas Search index (recommended)

Define a single index covering both vocabularies — the IIIF 3.0 paths (`body.value`, `bodyValue`, and the `items` / `annotations` embedded documents) and the IIIF 2.1 paths (`resource.chars`, `resource.cnt:chars`, and the `resources` / `otherContent` / `sequences` embedded documents).

`/search` becomes one `$search` with all the existing `should` clauses, followed by `$skip` and `$limit`:

- **Ranking becomes native.** Atlas returns `$search` results in descending score order across all clauses, and every document is scored once against one index rather than compared across two.
- **Deduplication disappears.** Atlas returns each document exactly once; there is no key to derive and nothing to drop. `mergeSearchResults()` can be deleted, and with it both bugs from #307.
- **Paging runs in the database**, so cost stops scaling with depth.

Cost: an Atlas-side index change, and both index generations must exist during the transition.

### 2. `$unionWith` in a single pipeline

Keep both indexes, run the IIIF 2.1 branch as a `$unionWith` sub-pipeline against the same collection, then `$group` on `_id`, `$sort` by score, and `$skip`/`$limit` — all server-side. Grouping on `_id` in Mongo compares structurally, so embedded-object ids group correctly there.

This moves the work off the API process but not off the cluster: the `$group` is a blocking stage over the union, so Atlas still materializes the merged set. Smaller change, smaller payoff. **Confirm `$search` is permitted as the first stage of a `$unionWith` sub-pipeline on our cluster tier before committing to this** — it is not universally available.

### 3. Keep the shape, bound the damage

If neither restructuring is scheduled soon, enforce a `/search`-specific `skip` maximum far below the `/query` maximum, since the two endpoints pay very different prices for the same depth. This is a mitigation, and it makes the two endpoints inconsistent — exactly what the shared-contract work argues against — so it should be a deliberate, documented decision rather than a default.

## Notes

- **Breaking.** Option 1 changes relevance scoring, and should: a single index scores a document once across all matched clauses. Today the two branches' scores are never compared against each other at all, so ordering will shift substantially. That is the point, not a regression. Membership should change only by the currently dropped documents reappearing.
- Depends on #308. Reviewing a new index definition against two that exist only in a browser tab is not a review.
- Should land after #307, which is the cheap correctness fix and is not blocked by anything. If option 1 ships, it supersedes both of that issue's fixes — but it should not gate them.
- `rel="next"` (#302) needs `/search` to over-fetch one record. Under the current shape that is `limit + skip + 1` per branch; under option 1 it becomes the ordinary single-pipeline `limit + 1`.
- Once this lands, `/search` can take a real cursor. Atlas Search's `searchAfter` / `searchSequenceToken` is the keyset equivalent for `$search` results — **confirm availability on our cluster tier and server version before planning it.** See #303 for why a cursor before this lands would be dishonest.
- Whatever lands must keep the property already verified: sequential pages equal to a single larger request, in the same order.

## Acceptance criteria

- [ ] An option is chosen and the reasoning is recorded on the issue before implementation
- [ ] `/search` pagination is applied in MongoDB rather than in application memory
- [ ] `/search` latency at depth is flat with respect to `skip`, comparable to `/query`, measured with the same probes used above
- [ ] Every document matched by the search is reachable by paging, including documents whose `_id` is an embedded object
- [ ] Results are ordered by descending relevance score, verified by asserting the score sequence never increases
- [ ] Sequential paged results still equal a single larger request, in the same order
- [ ] Relevance ordering change is measured on representative queries and accepted before cutover
- [ ] `/search` and `/search/phrase` both covered, and the unmounted search variants updated to match
- [ ] The ordering guarantee is stated in `openapi/contracts/core-provider.openapi.yaml`, not only in JSDoc

Start with controllers/search.js, especially buildDualIndexQueries(), mergeSearchResults(), and the /search and /search/phrase handlers; compare their pagination with controllers/crud.js:87. Inspect the index definitions from #308 and the contract in openapi/contracts/core-provider.openapi.yaml, then run the documented deep-page probes. Done means an agreed option is implemented and measured, with coverage for ordering, embedded-object IDs, sequential pages, both mounted endpoints, and the unmounted variants.

Written by the indexing model from the issue text.