Make paged reads on `/query` and `/search` honest and self-describing

**Detailed verification report.** This is the full investigation this issue and its sub-issues were written from.

---

## Summary

`/query` and `/search` accept `limit` and `skip`, clamp both to server maximums, and then return the clamped page with no indication that clamping happened.
A client cannot tell a full page from a truncated one, and cannot tell "no more results" from "you asked for more than I will give you."

Both endpoints share `getPagination()` (`controllers/utils.js:21`), so this is one contract with one fix site, not two. The same helper also paginates the two Gallery of Glosses endpoints (`controllers/gog.js:36,167`, default 50, JWT-gated), so they inherit whatever changes here. This issue covers that shared contract. Where `/search` performs its paging — in application memory rather than in MongoDB — is a separate concern, tracked in the `/search` paging issue.

Three distinct failures fall out of that silence:

- A client that stops when `results.length < limit` **silently drops records**.
- A client that stops when `results.length === 0` and advances by `results.length` **never terminates** once `skip` passes its maximum, because every subsequent request returns the same page.
- The deployment's configured maximums are **not in effect at all** — `.env` and the code disagree on the variable names.

The second failure is the one to look at first: the non-terminating client is the paged-query recipe published in our own API documentation (`public/API.html:555`), and the loop is reachable on production today with ordinary typed queries.

Every behavior below was verified against the public deployments `https://store.rerum.io` and `https://devstore.rerum.io`, not only locally.

## Why this matters

Pagination is a contract. The server promises that a client can walk a result set to completion by advancing `skip` until the results run out. Today the server breaks that promise in whichever direction the client happens to lean:

**A `length < limit` stop condition loses data.** The client asks for `limit=1000`, the server decides on 500, returns 500, and says nothing. `500 < 1000` reads as "last page." The walk terminates early and the client reports success on a partial result set. Nothing on the wire distinguishes this from a genuine final page. This is how the problem was found: a client with a configurable page size above 500 walked a result set, received 500, and stopped.

**A `length === 0` stop condition hangs.** Past the `skip` maximum every request returns an identical page. A client that concatenates and advances by `results.length` re-fetches that same page forever, growing its accumulator without bound. This is not a hypothetical client shape — it is the exact `pagedQuery` function we publish as the recommended pattern, and it is reachable on the production data set today.

**We already knew, and documented it as folklore.** `public/API.html` warns three times that "your application may experience strange behavior with large limits, such as ?limit=1000" (lines 507, 612, 730). The strange behavior is this truncation. A documented warning not to trigger a bug is not a fix, and nothing enforces it — a client sets its own `MAX_LIMIT` above the server's and the failure is silent from that point on.

**Clients have no way to discover the limits.** No response header carries the applied `limit`/`skip` or the server maximums. The only `Link` header emitted is the JSON-LD context (`utils.js:163`). `HEAD /query` returns `Content-Length` for the page it would have sent, not a total count (`controllers/history.js:86`), and it answers an empty page with a 404 where `POST /query` answers `200 []`, so it cannot be used to probe ahead either. `GET /v1/api` lists endpoints and nothing else, and the checked-in OpenAPI contract does not mention `limit` or `skip` at all. A well-written client that *wants* to defend itself cannot.

## Evidence

Verified 2026-09-02 against three deployments, all read-only `POST /query` and `POST /search` requests:

- **local** — the pm2 instance at `localhost:3001`, which reads the same collection as `devstore.rerum.io` (the `rerum-test` cluster, `annotationStore.alpha`)
- **devstore** — `https://devstore.rerum.io`
- **store** — `https://store.rerum.io`, production

Independently re-run twice on 2026-09-03, read-only, against `localhost:3001` for source lines and endpoint behavior, with the clamp, `skip`-repeat, and `HEAD` `Content-Length` checks re-run against `devstore.rerum.io` and `store.rerum.io`. Every reproducible number below held: the clamp values on all three deployments, the repeating page past `skip=100000` (`…4c0ef016` at `skip=99999`, `…4c0ef019` at `skip=100000` through `skip=500000` locally; `…645fb6b6` / `…645fb6b7` on production), the `HEAD /query` `Content-Length` repeat (1883 local, 1891 devstore, 1404 store), the production reachability table, every parsing edge case, and the tiling checks. The environment-variable fix was verified end-to-end: setting `RERUM_MAX_QUERY_LIMIT=10` / `RERUM_MAX_QUERY_SKIP=20` in the local `.env` and restarting pm2 made `limit=1000` return exactly 10 documents and `skip=20`, `25`, and `100` return the identical page; the `.env` change was then reverted and pm2 restarted again to restore the original 500 / 100000 caps. Two citations were corrected along the way: the published `pagedQuery` function in `public/API.html` begins at line 555, not 556, and the `getPagination()` test block is `__tests__/utils.test.js:329-354`, not 329-355.

### Limit is clamped silently, both endpoints, all deployments

| Requested `limit` | `/query` returned | `/search` returned |
|-------------------|-------------------|--------------------|
| 100 | 100 | 100 |
| 500 | 500 | 500 |
| 1000 | **500** | **500** |
| 5000 | **500** | **500** |

Identical on local, devstore, and store (`limit=1000` returns 500 on all three, both endpoints). Nothing in the response headers describes pagination: the clamped response carries `Allow`, `Content-Type`, `Content-Length`, an `ETag`, the JSON-LD context `Link`, and the CORS headers — no applied `limit` or `skip`, no maximum, no total.

### Skip is clamped to a fixed page that then repeats forever

Querying `{"__rerum.APIversion":{"$exists":true}}` with `limit=2` and comparing the first returned id:

| `skip` | local / devstore | store (production) |
|--------|------------------|--------------------|
| 9999 | `…e8255df7` | `…a7acc9f3` |
| 10000 | `…e8255f68` | `…a7acc9f4` |
| 10001 | `…e8255d5e` | `…a7acc9f5` |
| 99999 | `…4c0ef016` | `…645fb6b6` |
| 100000 | `…4c0ef019` | `…645fb6b7` |
| 100001 | **`…4c0ef019`** | **`…645fb6b7`** |
| 150000 | **`…4c0ef019`** | **`…645fb6b7`** |
| 500000 | **`…4c0ef019`** | — |

The effective `skip` maximum is 100000 on every deployment, including production. Consecutive pages at `skip=100000` and `skip=100100` with `limit=100` return byte-identical id lists (same md5 over the ids). Any client advancing past 100000 receives the same records indefinitely.

`HEAD /query` behaves identically: `skip=100000` and `skip=150000` report the same `Content-Length` for the same repeated page on every deployment (1883 local, 1891 devstore, 1404 store).

`/search` applies the same clamp, but it slices the page from its merged in-memory set (see the `/search` issue), so the repeated page only appears for a term with more than 100000 merged matches. No term tried on the dev collection reaches that — `line`, `the`, and `of` all return an empty page at `skip=100000`, `100100`, and `150000` — so the documented client terminates on `/search` today. The mechanism is identical; only the data keeps it out of reach.

### The loop is reachable on production with realistic queries

Probing `store.rerum.io` with `limit=1` at increasing `skip`:

| Query | `skip=10000` | `skip=50000` | `skip=100000` |
|-------|--------------|--------------|---------------|
| `{"@type":"oa:Annotation"}` | 1 | 1 | **1** |
| `{"__rerum.generatedBy":{"$exists":true}}` | 1 | 1 | **1** |
| `{"__rerum.APIversion":{"$exists":true}}` | 1 | 1 | **1** |
| `{"@type":"Annotation"}` | 1 | 1 | 0 |
| `{"type":"Annotation"}` | 0 | 0 | 0 |
| `{"type":"Thing"}` | 0 | 0 | 0 |

A document at `skip=100000` means the query has more matches than the cap. `{"@type":"oa:Annotation"}` — every IIIF 2.1 annotation — is over the cap on production right now. `{"@type":"Annotation"}` is between 50000 and 100000 and growing toward it.

### The documented client does not terminate

The published `pagedQuery` from `public/API.html:555` was run verbatim — only a request counter, an iteration guard, and logging added — against the local deployment at the real caps (500 / 100000).

Query `{"type":"Thing"}` with `limit=100`, 29 matching documents. Terminates correctly:

```text
req #1: skip=0 -> 29 docs, first=6aab775e
req #2: skip=29 -> 0 docs
=> 29 records, 29 distinct
```

Query `{"__rerum.APIversion":{"$exists":true}}` with `limit=100`, entered at `skip=99800` so the cap is reached cheaply. Never terminates:

```text
req #1: skip=99800 -> 100 docs, first=912509cc
req #2: skip=99900 -> 100 docs, first=4c0eef00
req #3: skip=100000 -> 100 docs, first=4c0ef019
req #4: skip=100100 -> 100 docs, first=4c0ef019
req #5: skip=100200 -> 100 docs, first=4c0ef019
req #6: skip=100300 -> 100 docs, first=4c0ef019
req #7: skip=100400 -> 100 docs, first=4c0ef019
req #8: skip=100500 -> 100 docs, first=4c0ef019
GUARD TRIPPED after 8 requests
=> 800 records accumulated, 300 distinct
```

From request #3 on, every page is the same page. Without the guard this does not stop and the accumulator grows without bound. The same run against a deployment with the caps temporarily lowered to `limit=10` / `skip=20` shows the same shape from `skip=20` on: 12 requests, 110 records accumulated, 29 distinct.

Note that the published example never terminates on this query at *any* page size, because its only stop condition is an empty page. At `limit=1000` it receives 500-record pages, advances by 500, reaches the cap after 200 requests, and then repeats forever exactly as above.

The other common client shape fails the other way. Running the same function with its stop condition changed to `page.length < limit` — the shape the reporting client used — at `limit=1000` against the same query:

```text
req #1: skip=0 -> 500 docs
=> 500 records (client believes the walk is complete)
```

`500 < 1000`, so the client stops and reports a complete walk of 500 records. The query matches more than 100000.

### The configured maximums are inert

`.env` in this working tree (the deploy workflows write the servers' `.env` from the `DEV_FULL_ENV` / `PROD_FULL_ENV` secrets in the same shape) sets:

```text
MAX_QUERY_LIMIT=500
MAX_QUERY_SKIP=10000
```

The code reads different names (`controllers/utils.js:12-13`):

```js
const MAX_QUERY_LIMIT = Number.parseInt(process.env.RERUM_MAX_QUERY_LIMIT ?? 500, 10)
const MAX_QUERY_SKIP = Number.parseInt(process.env.RERUM_MAX_QUERY_SKIP ?? 100000, 10)
```

`env-loader.js` loads keys verbatim and does no prefixing, so neither `RERUM_`-prefixed variable is ever set and both fall back to the code defaults. `limit` appears configured only because 500 happens to be both the intended value and the default.

Confirmed three ways:

- Both public deployments clamp at exactly the code defaults, 500 and 100000 (tables above). If `MAX_QUERY_SKIP=10000` is set on either server, it is being ignored. If a configured `skip` cap of 10000 were honored, `skip=9999`, `skip=10000` and `skip=10001` would return the same page; on every deployment they return three different pages.
- Adding the prefixed names to the local `.env` and restarting takes effect immediately: `RERUM_MAX_QUERY_LIMIT=10` / `RERUM_MAX_QUERY_SKIP=20` makes `limit=1000` return 10 documents and `skip=20`, `25`, `100` return an identical page. This both proves the diagnosis and validates the rename as a fix.
- The existing unit tests for `getPagination()` (`__tests__/utils.test.js:329-354`) assert only that a huge `limit` is clamped "below `Number.MAX_SAFE_INTEGER`". That passes for any cap under any variable name, which is why the mismatch was never caught. Nothing documents the keys either: `.env` is not tracked and the repository has no `.env` template, the #262 description that introduced them does not name them, and they appear in neither `CONTRIBUTING.md`, `README.md`, nor `.github/copilot-instructions.md`.

The operator-visible consequence: someone tightened `skip` to 10000 in `.env` and the server has been allowing 100000 ever since.

### Parsing edge cases

Both endpoints, from `clampNonNegativeInt()` (`controllers/utils.js:15`). Every one of these returns 200:

```text
limit=0 -> 100 limit=abc -> 100 limit=10abc -> 10
limit=-5 -> 100 limit=1e3 -> 1 limit=0x10 -> 100
limit=250.7 -> 250 limit= -> 100

skip=-5 -> 0 skip=abc -> 0 skip=1e3 -> 1
skip=2.9 -> 2 skip= -> 0
```

`limit=1e3` is the worst of these: `Number.parseInt("1e3", 10)` is `1`, so the client receives a single record. A `length < limit` client stops after one object and reports a completed walk. The same parse makes `skip=1e3` land at offset 1 rather than 1000. Non-numeric and non-positive values silently become the defaults rather than being rejected.

A repeated parameter is silently reduced to its first value, from the same `parseInt` call. Express 5's `simple` query parser (the default this app uses — `app.set('query parser')` is never called) hands `getPagination()` an array, and `Number.parseInt(["100","200"], 10)` coerces it to the string `"100,200"` and parses `100`:

```text
?limit=100&limit=200 -> 100 records
?limit=200&limit=100 -> 200 records
?skip=0&skip=1000&limit=1 -> offset 0
```

A bracketed parameter also silently becomes the default: `?limit[a]=5` returns 100 records, because the `simple` parser keeps `limit[a]` as a literal key and `req.query.limit` is never set. Under the `extended` parser it would instead arrive as the object `{a:"5"}`, `parseInt` would give `NaN`, and the fallback would produce the same 100 — so this one is insensitive to the parser choice, but only by accident. Neither case is an error today; both are cases where a client's intent and the server's behavior differ with nothing on the wire to say so.

### Documented default is wrong

`public/API.html:506` states the response is "limited to 10 records" by default. The actual default is 100 (`getPagination(req.query, 100)`, called at `controllers/crud.js:77` and `controllers/search.js:274`). A request with no `limit` parameter returns 100.

### The machine-readable contract does not describe pagination

`openapi/contracts/core-provider.openapi.yaml` is the checked-in contract for this API — synced to the shared spec repository by `.github/workflows/sync-rerum-shared-openapi.yml` and guarded by `__tests__/openapi_sync_artifacts.test.js`. It defines `/api/query` (POST and HEAD), `/api/search`, and `/api/search/phrase`, and **none of the four declares a `limit` or `skip` parameter**. The words do not appear anywhere in the file, and neither does any pagination-related response header. The only reusable query parameters it defines are `ExpansionGenerator` and `ExpansionCreator`, for `/id/{id}/expanded`.

So a client generated from the contract cannot page at all, and a client written by hand against the contract has no reason to believe paging exists. Whatever shape the fix takes, the parameters, their maximums, and any `Link` header need to land here as well as in `public/API.html` — otherwise the honest behavior is still undiscoverable through the artifact that is supposed to be authoritative.

### `HEAD /query` is not a count

`HEAD /query?limit=2` returns `Content-Length: 11832`, which is exactly the byte size of the body `POST /query?limit=2` returns. `limit=100` gives 68396 and `limit=1000` gives 949640 (the clamped 500-record page). It reports the size of one page, subject to the same clamping, and says nothing about how many records match.

It also disagrees with `POST` about what the end of a result set looks like. For the same request body and the same pagination parameters:

```text
POST /query?limit=2 {"type":"NoSuchTypeAtAll"} -> 200, []
HEAD /query?limit=2 {"type":"NoSuchTypeAtAll"} -> 404

POST /query?limit=2&skip=100 {"type":"Thing"} -> 200, [] (29 matches, so this is past the end)
HEAD /query?limit=2&skip=100 {"type":"Thing"} -> 404
```

`queryHeadRequest()` treats an empty page as "no objects in the database matching the query" (`controllers/history.js:99-101`) rather than as an exhausted page. So the one endpoint that could cheaply tell a client where a result set ends signals that with a status code the paged endpoint never uses, and a client cannot use `HEAD` to probe ahead of `POST` without special-casing the mismatch. A HEAD response is also supposed to carry the headers its GET/POST counterpart would send; a 404 against a 200 is not that. #96 already questions whether these HEAD handlers should exist at all.

### What is working correctly

`/search` pagination *tiles* consistently — five sequential `limit=100` pages return exactly the same records in the same order as a single `limit=500` request, and the tail terminates (on the dev collection, searching `line`: `skip=3700` returns 100, `skip=3800` returns 25, `skip=3900` returns 0). Page boundaries and membership are stable, which is what this issue is about.

`/query` tiles as well: five sequential `limit=100` pages from `skip=50000` return the same 500 ids in the same order as one `limit=500` request, and repeating a request returns the same page. One caveat for the fix: `/query` applies no `sort`, so page boundaries rest on MongoDB natural order. That holds in practice because RERUM marks deletions rather than removing documents, but it is not a guarantee the server makes, and a `rel="next"` implementation should sort on `_id` so that consecutive pages are deterministic by construction.

That is not the same as `/search` being correct. It pages in application memory, its results are not ordered by relevance, and its merge step silently drops some matching documents before paging even begins — so the "honest" tail is honest only relative to a pool that has already lost records. All three are covered in the separate `/search` issue.

## Affected lines

| File | Line | Current |
|------|------|---------|
| `controllers/utils.js` | 12-13 | Reads `RERUM_MAX_QUERY_LIMIT` / `RERUM_MAX_QUERY_SKIP`; `.env` sets the unprefixed names |
| `controllers/utils.js` | 15-19 | `clampNonNegativeInt()` silently substitutes fallbacks and caps |
| `controllers/utils.js` | 21-28 | `getPagination()` returns clamped values with no report of clamping |
| `controllers/crud.js` | 77, 87 | `/query` paging |
| `controllers/history.js` | 89, 91 | `HEAD /query` paging; no total count |
| `controllers/history.js` | 99-103 | `HEAD /query` returns 404 on an empty page where `POST /query` returns `200 []` |
| `controllers/gog.js` | 36, 167 | `/gog/fragmentsInManuscript` and `/gog/glossesInManuscript` paging, default 50; same helper, same clamping |
| `controllers/search.js` | 274, 282 | `/search` paging |
| `controllers/search.js` | 360, 368 | `/search/phrase` paging |
| `utils.js` | 163 | `configureLDHeadersFor()` emits only the context `Link` |
| `__tests__/utils.test.js` | 329-354 | `getPagination()` tests pin neither the configured cap nor the env key names |
| `openapi/contracts/core-provider.openapi.yaml` | 210, 233, 241, 267 | `/api/query`, `HEAD /api/query`, `/api/search`, `/api/search/phrase` declare no `limit` or `skip` parameter |
| `public/API.html` | 506 | Documents a default of 10; actual default is 100 |
| `public/API.html` | 507, 612, 730 | Warns about "strange behavior" instead of the server being honest |
| `public/API.html` | 555 | Published `pagedQuery` example is the non-terminating shape |

## Proposed change

The goal is that a correct client never has to guess. Ordered by how much data loss each item prevents.

### 1. Reject an out-of-range `skip` instead of clamping it

Silently returning a page the client did not ask for, forever, has no defensible reading. Return 400 naming the maximum:

```json
{
"message": "The skip value 150000 exceeds the maximum of 100000.",
"status": 400
}
```

This alone converts the non-terminating loop into an immediate, legible failure.

### 2. Advertise the next page with a `Link` header

`Link: <…>; rel="next"` (RFC 8288) when another page exists, omitted when the result set is exhausted. Clients follow `next` and stop when it is absent, which makes every length-based stop condition irrelevant.

The server can know this cheaply by over-fetching one record and trimming it before serialization:

- `/query`: `db.find(props).limit(limit + 1).skip(skip)`
- `/search`: `$limit: limit + skip + 1` per branch, then slice as today

`Access-Control-Expose-Headers: *` is already set, so browser clients can read it. Note that `configureLDHeadersFor()` currently assigns `Link` as a single string — `rel="next"` needs to be appended to the existing context link rather than replacing it.

### 3. Make an over-maximum `limit` visible

Either reject it with a 400 the way `skip` is handled above, or keep clamping as a safety valve and report the applied value in a response header. Clamping is reasonable server behavior; doing it in silence is not. Rejecting is the more honest of the two and is the recommendation, though it is the more breaking of the two — worth a decision on this thread.

### 4. Fix the environment variable names

Either rename the keys in `.env` on both deploy servers to the `RERUM_`-prefixed form, or change `controllers/utils.js` to read the unprefixed names. Whichever way, the two must agree, `CONTRIBUTING.md` should document the keys alongside the rest of the `.env` contents, and a unit test should set the key and assert the cap it produces — the current tests would pass under either name. Reading both names with the prefixed one winning would avoid a coordinated deploy, if that is preferred.

### 5. Reject unparseable and non-positive `limit`

`limit=abc`, `limit=-5`, and `limit=0` should be 400s, not a silent 100. `limit=1e3` (and `skip=1e3`) should either parse as 1000 or be rejected; returning one record is the worst available option. A repeated parameter (`?limit=100&limit=200`) should be rejected too — `Number.parseInt` currently coerces the array to a comma-joined string and takes the first value, which is a guess dressed up as a result. Validating the raw parameter as a decimal integer string, and rejecting anything that is not a string, covers all of these in one place. `?limit[a]=5` is the one case that cannot be caught: under the `simple` parser the server never receives a `limit` key at all, so it is indistinguishable from a request that omitted the parameter.

### 6. Publish the limits

Add the effective maximums and default to the `GET /v1/api` response so clients can configure themselves against the deployment they are actually talking to rather than a hardcoded guess. Today that response is a plain list of endpoint descriptions and carries no pagination information.

### 7. Declare `limit` and `skip` in the OpenAPI contract

Add a shared `PageLimit` / `PageSkip` parameter pair under `components/parameters` in `openapi/contracts/core-provider.openapi.yaml` and reference it from `/api/query` (POST and HEAD), `/api/search`, and `/api/search/phrase`, with the maximums in the schema and a 400 response for out-of-range values once items 1, 3, and 5 land. If `rel="next"` ships, describe the `Link` response header there too. Without this the contract keeps describing an API that has no pagination.

### 8. Make `HEAD /query` agree with `POST /query`

An empty page is not a 404. `HEAD /query` should return 200 with `Content-Length: 2` (the `[]` body `POST` would send) so that a client can probe with either verb and read the same signal. This is a small change inside `queryHeadRequest()`, and it is worth resolving alongside #96 rather than independently of it.

## Notes

- Fixing `getPagination()` covers `/query`, `HEAD /query`, `/search`, `/search/phrase`, and the two `/gog/*InManuscript` endpoints together, which is what keeps them consistent by construction. The `rel="next"` work is the only part needing per-controller changes.
- `public/API.html` needs updating in the same PR: correct the documented default from 10 to 100, replace the "strange behavior" warnings with the actual maximums, and revise the `pagedQuery` example to follow `rel="next"` (or at minimum to stop on a short page rather than only on an empty one).
- Items 1, 3, 5, and 8 change response codes for requests that currently return 200 (or, for 8, a 404 that becomes a 200). That is a breaking change for any client relying on the clamping, and it should land on dev first with a note to known client maintainers.
- The two issues compound in one place worth noting here: because `/search` pages in application memory, the `skip` maximum is also what bounds how many documents that endpoint will pull into the API process for a single request. Lowering the cap helps both issues.
- History: `getPagination()` and the caps arrived in #262 ("251 memory reins", merged 2026-05-07) as the fix for #251; that PR's description standardizes pagination across the CRUD, search, history, and Gallery of Glosses controllers but never names the environment variables. #252 is still open: its first recommendation (a default and maximum `limit` on query endpoints) is what #262 shipped, and its third (cursor-based pagination) is the `rel="next"` work proposed here; its `getAllVersions` and `/history` items are separate. #96 questions whether `HEAD /query` should exist.

## Acceptance criteria

- [ ] `skip` above the maximum returns 400 rather than a repeated page
- [ ] `limit` above the maximum is either rejected or reported in a response header — never silently applied
- [ ] Responses carry `Link: …; rel="next"` while more results exist, and omit it on the final page
- [ ] `limit=abc`, `limit=-5`, and `limit=0` return 400
- [ ] `.env` keys and `controllers/utils.js` agree on the maximum names, verified by observing the configured cap take effect on devstore
- [ ] A unit test sets the maximum via the environment and asserts the resulting cap
- [ ] `/query`, `/search`, and the other `getPagination()` callers (`HEAD /query`, `/gog/*InManuscript`) behave identically for every case above
- [ ] `public/API.html` documents the real default, the real maximums, and a paged example that terminates correctly
- [ ] A repeated `limit` or `skip` parameter is rejected rather than silently reduced to its first value
- [ ] `HEAD /query` and `POST /query` return the same status for the same request, including on an empty page
- [ ] `openapi/contracts/core-provider.openapi.yaml` declares `limit` and `skip` on `/api/query`, `HEAD /api/query`, `/api/search`, and `/api/search/phrase`, with the maximums and any new response codes
- [ ] Regression tests cover the clamp boundaries and the `rel="next"` presence and absence