`/search` misranks results, drops matching documents, and pages in application memory

**Detailed verification report.** This is the full investigation this issue and its sub-issues were written from.
---

## Summary

`/search` fans out to two Atlas Search indexes as two separate `db.aggregate()` calls, then merges, deduplicates, sorts, and slices the combined result in Node (`controllers/search.js:282`). Three problems follow from that design, two of them user-visible:

1. **Relevance ranking does not work at all.** The merge sorts on a field that does not exist, so the comparator returns `0` for every pair and the sort is a no-op. Results come back as "every IIIF 3.0 hit, then every IIIF 2.1 hit," regardless of score. On production the single best-scoring match for `line` — scoring 84 — is ranked 6th, behind five matches scoring under 2.
2. **The deduplication step drops matching documents.** The merge keys each document by `_id?.$oid || _id?.toString()`. Legacy documents whose `_id` is an embedded object (the v0 import from `165.134.105.29/annotationstore`) all key as the string `"[object Object]"`, so the first one wins and every other one is silently discarded. On the dev collection, 182 of the 4007 documents matching `line` are unreachable through `/search` on any page.
3. **Paging costs scale with depth.** `/query` pushes pagination down to Mongo — `db.find(props).limit(limit).skip(skip)` (`controllers/crud.js:87`). `/search` cannot, so every page costs the server everything up to that page, on every request.

Nothing about `$search` requires any of these. A single Atlas Search pipeline sorts by score natively, returns each document once, and can end in `$skip`/`$limit`. The blocker is the two-index fan-out, not the search stage — which is why one change fixes all three.

The pagination *contract* problems shared by both endpoints — silent clamping, the repeating page past the `skip` maximum, the inert environment variables — are tracked separately in the paged-reads issue. This issue is about `/search`'s own internals.

## Why this matters

### The ranking bug

`buildDualIndexQueries()` writes each document's relevance score to `__rerum.score` (`controllers/search.js:210,215`):

```js
{ $addFields: { "__rerum.score": { $meta: "searchScore" } } }
```

`mergeSearchResults()` then sorts on a top-level `score` (`controllers/search.js:45`):

```js
return merged.sort((a, b) => (b.score || 0) - (a.score || 0))
```

No document has a top-level `score`. Every comparison evaluates `0 - 0`, the sort does nothing, and the merged array keeps its construction order — `[...results1, ...results2]`, IIIF 3.0 first. The JSDoc on both search handlers promises results "sorted by relevance score (highest first)". They are not, and have not been.

This is not subtle in practice. On production, searching `line` returns five IIIF 3.0 matches scoring between 1.69 and 2.25, and then at index 5 the score **jumps to 83.9** — the IIIF 2.1 boundary and the best match in the entire set, ranked behind five documents that score fifty times lower. On the dev collection, where the IIIF 3.0 index has 205 hits for the same term, the best match (score 12.0) is ranked 206th; a client showing "top 10 results" never sees it, and a client paging at 100 per page finds it on page 3.

Any deployment where one vocabulary dominates the corpus will see the other vocabulary's matches effectively buried, no matter how well they score.

### The dropped documents

`mergeSearchResults()` (`controllers/search.js:37`) derives its deduplication key as:

```js
const id = result._id?.$oid || result._id?.toString()
```

That is correct for an `ObjectId` and for a string. It is wrong for a plain object: `({}).toString()` is `"[object Object]"`. The dev collection holds **6427** documents whose `_id` is an embedded object — the serialized Java `ObjectId` shape (`{date, inc, machine, time, …}`) left behind by the v0 import. Every one of them that matches a search collapses onto the same key, the first is kept, and the rest are dropped before paging begins.

Measured directly against the two branch pipelines on the dev collection at full depth (`$limit` 100100 per branch):

| Term | `presi3` raw | `presi2` raw | Cross-index overlap | Embedded-object `_id` | Merged | **Dropped** |
|------|--------------|--------------|---------------------|-----------------------|--------|-------------|
| `line` | 205 | 3802 | 0 | 183 | 3825 | **182** |
| `text` | 2301 | 46 | 0 | 4 | 2344 | **3** |
| `manuscript` | 0 | 68 | 0 | 19 | 50 | **18** |
| `page` | 63 | 3 | 0 | 0 | 66 | 0 |

Two consequences for a paging client:

- **Records are unreachable.** No `skip` value ever returns the 182 dropped `line` documents. A client that walks `/search` to the empty page believes it has everything.
- **Pages under-fill before results are exhausted.** `manuscript` with the default `limit=100` and `skip=0` returns 50 documents from 68 matches. A `length < limit` client concludes there are 50 results and stops. This is the hazard the per-branch `$limit`-before-dedup shape always had; it is not hypothetical.

Cross-index overlap, by contrast, really is zero for every term tried — a document is written in either IIIF 3.0 or IIIF 2.1 shape, and the two indexes cover disjoint field paths. The deduplication is not doing the job it was written for; it is doing a different, harmful one.

Production currently has **no** embedded-object `_id` documents (`{"_id.inc":{"$exists":true}}` returns nothing on `store.rerum.io`), so the drop is not observable there today. It is a defect in the code path regardless: any document whose `_id` is neither an `ObjectId` nor a string will be lost the same way, and the dev deployment is where clients are developed against.

### The paging cost

**Cost scales with depth, not page size.** Each branch caps at `$limit: limit + skip` (`controllers/search.js:211,216`), so a request for page 40 makes Atlas produce and return 4000 documents per index, and makes Node sort and deduplicate up to 8000 of them, in order to hand back 100. `/query` at the same depth asks Mongo to skip and returns 100.

**The ceiling is set by the `skip` maximum.** With the effective maximum at 100000 (see the paged-reads issue — the configured 10000 is not in effect on any deployment), a sufficiently broad search term at maximum depth asks Node to hold on the order of 100000 documents per branch in memory and sort them. Full RERUM documents, not identifiers. The two issues compound: the cap that is too high is also the cap that bounds this blast radius.

**It is a shared-process cost.** This work happens in the API process, not in Atlas. Under PM2 cluster mode a few concurrent deep searches contend for the same workers that serve every other endpoint.

## Evidence

Verified 2026-09-02. "Dev" means the local pm2 instance at `localhost:3001`, which reads the same collection as `devstore.rerum.io` (the `rerum-test` cluster, `annotationStore.alpha`); the same numbers were confirmed on `devstore.rerum.io` where noted. "Prod" means `https://store.rerum.io`. All probes are read-only.

Independently re-run twice on 2026-09-03, read-only, against `localhost:3001` and directly against the `presi3AnnotationText` / `presi2AnnotationText` indexes (bypassing the endpoint to isolate the merge step), with the score-ordering check also re-run against `devstore.rerum.io` and `store.rerum.io`. Every number reproduced: the `line`/`text`/`manuscript`/`page` raw and dropped counts, the score jump for `line` at index 205 (1.958 → 12.013, rank 206th of 500 returned) on both local and devstore, the identical boundary on `/search/phrase`, the `manuscript` under-fill (68 raw matches, 50 returned at `limit=100`), the empty pages for `line` at `skip=100000/100100/150000`, the 6427 embedded-object `_id` documents on dev against none on production, and the tiling and latency-scaling shapes. No document keyed on any embedded-object `_id` was found to collide with a distinct document's key by anything other than the `"[object Object]"` collision already described — the mechanism is exactly as diagnosed. One number moved: production's best `line` score now reads 83.928 rather than 83.925, which is ordinary drift as the corpus and index change and does not affect the rank.

### Ranking is broken, measured through the endpoint

The results of a single `/search?limit=500` request, read back via `.__rerum.score`. A correctly sorted list never increases.

| Deployment | Term | Returned | Score increases at | Values at the increase | Best score in set | Rank of best |
|------------|------|----------|--------------------|------------------------|-------------------|--------------|
| prod | `line` | 500 | index 5 | 1.690 → 83.928 | 83.928 | 6th |
| prod | `text` | 24 | index 2 | 2.452 → 6.958 | 6.958 | 3rd |
| prod | `page` | 298 | index 5 | 1.843 → 3.971 | 3.971 | 6th |
| dev, devstore | `line` | 500 | index 205 | 1.958 → 12.013 | 12.013 | 206th |

In every case there is exactly one increase, and it sits exactly at the boundary between the two branches. `/search/phrase` shows the identical boundary (dev, `line`: index 205, 1.958 → 12.013). Confirmed at the source by running the pipelines directly: every returned document has `__rerum.score` populated and top-level `score` `undefined`, so `(b.score || 0) - (a.score || 0)` is `0` for every pair.

### Documents are dropped by the merge

The table in "The dropped documents" above. Confirmed by grouping raw branch output by the merge's own key: for `manuscript`, 68 raw results yield 50 distinct keys, and the one colliding key — `"[object Object]"` — has 19 members, each a different document with a different `_id`, all `@type: oa:Annotation` from the v0 import.

Confirmed a second way, end to end through the endpoint rather than against the pipelines. Walking `/search` for `line` at `limit=500` until it returns an empty page reaches exactly **3825** records — the merged count, not the 4007 raw matches. The 182 dropped documents are not on a later page; they are not on any page:

```text
POST /v1/api/search?limit=500&skip=0 … until empty -> 3825 records total
raw branch matches (205 presi3 + 3802 presi2) -> 4007
```

The under-fill is visible in a single request with no walking at all:

```text
POST /v1/api/search?limit=100&skip=0 {"searchText":"manuscript"}
-> 50 documents (100 requested, 68 actually match)
```

A `length < limit` client stops right there and reports 50 of 68.

### Latency scales with depth

Dev, `{"searchText":"line"}`, `limit=100`:

```text
skip=0 -> 100 documents in 379ms
skip=1000 -> 100 documents in 664ms
skip=2000 -> 100 documents in 781ms
skip=3000 -> 100 documents in 1172ms
skip=3700 -> 100 documents in 2021ms
skip=3800 -> 25 documents in 1954ms
skip=3900 -> 0 documents in 1784ms
skip=5000 -> 0 documents in 1802ms
skip=20000 -> 0 documents in 1815ms
```

`/query` at the same depths for contrast, `{"__rerum.APIversion":{"$exists":true}}`, `limit=100`:

```text
skip=0 -> 100 documents in 340ms
skip=1000 -> 100 documents in 296ms
skip=5000 -> 100 documents in 323ms
skip=20000 -> 100 documents in 369ms
skip=99000 -> 100 documents in 445ms
```

The two seconds spent to return nothing at `skip=5000` is the cost of producing and merging the entire matching set. Cost plateaus once the result set is exhausted, which confirms the bound is `min(total matches, limit + skip)` per branch — for a broad term on a large collection, that bound is the `skip` maximum. Prod shows the same shape at a smaller scale. `line` has 574 prod matches: `skip=0` returns 100 in 621 ms, `skip=500` returns the last 74 in 941 ms, and the empty pages at `skip=1000`, `2000`, and `20000` still cost roughly 850–900 ms each.

### Paging is self-consistent, even though the ordering is wrong

Five sequential `limit=100` pages return exactly the same 500 records in exactly the same order as a single `limit=500` request. The tail terminates: `skip=3700` returns 100, `skip=3800` returns 25, `skip=3900` returns 0 — 3825 records reachable, which is exactly the merged count after the 182 drops.

Worth being precise about why: each branch returns a deterministic prefix of its own result set, the caps grow with `skip`, and the no-op sort leaves the concatenation order stable. So pages tile correctly — membership and page boundaries are stable. They are tiling a list that is in the wrong order and that is missing documents, and no client can page its way to either the best match or the dropped ones.

Past the `skip` maximum, `/search` returns an empty page rather than the repeated page described in the paged-reads issue, because the slice runs off the end of the merged set: `line`, `the`, and `of` all return 0 documents at `skip=100000`, `100100`, and `150000` on dev. The repeated page would need a term with more than 100000 merged matches, which no term tried reaches.

## Affected lines

| File | Line | Current |
|------|------|---------|
| `controllers/search.js` | 82 | `buildDualIndexQueries()` builds two independent pipelines |
| `controllers/search.js` | 84, 153 | Two separate indexes: `presi3AnnotationText`, `presi2AnnotationText` |
| `controllers/search.js` | 211, 216 | `$limit: limit + skip` per branch — the depth cost |
| `controllers/search.js` | 210, 215 | Score written to `__rerum.score` |
| `controllers/search.js` | 37 | Dedup key `_id?.$oid \|\| _id?.toString()` collapses embedded-object `_id` values to `"[object Object]"` — **the drop bug** |
| `controllers/search.js` | 45 | Sorts on top-level `score`, which is never set — **the ranking bug** |
| `controllers/search.js` | 32 | `mergeSearchResults()` deduplicates in Node, after the per-branch cap |
| `controllers/search.js` | 230, 238, 300, 386, 464, 557 | JSDoc promising results "sorted by relevance score" |
| `controllers/search.js` | 282, 368 | `merged.slice(skip, skip + limit)` — pagination in application memory |
| `controllers/search.js` | 642, 660 | Same shape hand-rolled in `searchAlikes()` |
| `controllers/crud.js` | 87 | `/query` for contrast: `.limit(limit).skip(skip)` in Mongo |
| `openapi/contracts/core-provider.openapi.yaml` | 241, 267 | `/api/search` and `/api/search/phrase` declare no `limit` or `skip` parameter and no ordering guarantee |

Live endpoints are `searchAsWords` (`:263`) and `searchAsPhrase` (`:346`), mounted at `/search` and `/search/phrase` (`routes/search.js:6,13`). `searchFuzzily` (`:420`), `searchWildly` (`:510`), and `searchAlikes` (`:614`) have the same structure but no routes yet — worth fixing together so the pattern is not carried forward when they are mounted.

## Proposed change

Three options, best first.

### 1. One combined Atlas Search index (recommended)

Define a single index covering both vocabularies — the IIIF 3.0 paths (`body.value`, `bodyValue`, and the `items` / `annotations` embedded documents) and the IIIF 2.1 paths (`resource.chars`, `resource.cnt:chars`, and the `resources` / `otherContent` / `sequences` embedded documents).

`/search` then becomes one `$search` with all the existing `should` clauses, followed by `$skip` and `$limit`. This fixes all three problems at once:

- **Ranking** becomes native. Atlas returns `$search` results in descending score order across all clauses, so there is no cross-branch merge to get wrong.
- **Dropped documents** cannot happen. Atlas returns each matching document exactly once; there is nothing to deduplicate and no key to derive.
- **Paging** runs in the database, so cost stops scaling with depth.

`mergeSearchResults()` can be deleted, and with it both bugs.

Cost: an Atlas-side index change, and both indexes must exist during the transition.

### 2. `$unionWith` in a single pipeline

Keep both indexes, run the IIIF 2.1 branch as a `$unionWith` sub-pipeline against the same collection, then `$group` on `_id` to deduplicate, `$sort` by score, and `$skip`/`$limit` — all server-side. Grouping on `_id` in Mongo compares the value structurally, so embedded-object ids group correctly there.

This removes the work from the API process but not from the cluster: the `$group` is a blocking stage over the union, so Atlas still materializes the merged set. It is a smaller change than option 1 with a smaller payoff. Before committing to it, confirm that `$search` is permitted as the first stage of a `$unionWith` sub-pipeline on our cluster tier.

### 3. Keep the current shape, bound the damage

If neither restructuring is scheduled soon, the depth cost can at least be capped by enforcing a `/search`-specific maximum on `skip` that is far below the `/query` maximum, since the two endpoints pay very different prices for the same depth. This is a mitigation, not a fix, and it makes the two endpoints inconsistent — which is exactly what the paged-reads issue argues against — so it should be a deliberate, documented decision rather than a default.

## Notes

- The index definitions are not in this repository. They exist only in the Atlas UI, which means the current search behavior is not reviewable or reproducible from source. Whichever option is chosen, the definitions should be checked in — that is arguably worth doing first, independently of this issue.
- The checked-in OpenAPI contract (`openapi/contracts/core-provider.openapi.yaml`) describes `/api/search` and `/api/search/phrase` as taking a body and returning a `GenericArray`, with no `limit` or `skip` parameter and no statement about ordering. Whatever ordering guarantee comes out of this work should be written there, not only in the JSDoc — the paged-reads issue covers the parameters themselves.
- Option 1 changes relevance scoring, and should. A single index scores a document once across all matched clauses and sorts natively. Today the two branches' scores are never compared at all, so ordering will change substantially — that is the point, not a regression. Membership should change only by the documents that are currently dropped reappearing. Worth a before-and-after comparison on representative queries so the shift is understood before cutover.
- Two one-line mitigations are available immediately, independent of any restructuring, and are worth landing first regardless of which option is chosen for the paging work:
- sort on `__rerum.score` instead of `score` in `mergeSearchResults()`, which makes cross-index ordering real without touching the index layout;
- derive the dedup key from something that is unique for every `_id` shape — `JSON.stringify(result._id)`, or the document's `@id` / `id` — instead of `toString()`.
Both change results for existing clients (order for everyone; membership on dev), so they are behavioral even though the diffs are trivial.
- The `rel="next"` work proposed in the paged-reads issue needs `/search` to over-fetch one record per branch (`limit + skip + 1`) to know whether another page exists. That applies under any of the three options here, and under option 1 it becomes the ordinary single-pipeline `limit + 1`.
- Whatever lands must keep the property verified above: sequential pages equal to a single larger request, in the same order.

## Acceptance criteria

- [ ] `/search` results are ordered by descending relevance score across both vocabularies, verified by asserting the returned score sequence never increases
- [ ] The JSDoc promise of relevance ordering is either true or removed
- [ ] Every document matched by the search is reachable by paging: the number of records a client can walk to equals the number of raw matches, including documents whose `_id` is an embedded object
- [ ] A regression test feeds `mergeSearchResults()` (or its replacement) documents with `ObjectId`, string, and embedded-object `_id` values and asserts none are dropped
- [ ] `/search` pagination is applied in MongoDB rather than in application memory
- [ ] `/search` latency at depth is flat with respect to `skip`, comparable to `/query`
- [ ] Search index definitions are checked into the repository
- [ ] Sequential paged results still equal a single larger request, in the same order
- [ ] Relevance ordering change is measured and accepted before cutover
- [ ] `/search` and `/search/phrase` both covered; the unmounted search variants updated to match
- [ ] The ordering guarantee is stated in `openapi/contracts/core-provider.openapi.yaml`, not only in JSDoc

Start in controllers/search.js, especially mergeSearchResults() around line 37 and buildDualIndexQueries() around lines 210-216, then reproduce the reported /search probes for ranking, dropped documents, and deep paging. Done means results are relevance-ordered, embedded-object IDs are not collapsed, and pagination no longer requires merging and sorting all preceding matches in application memory.

Written by the indexing model from the issue text.