CenterForDigitalHumanities / CenterForDigitalHumanities/rerum_server_nodejs
Paged responses carry no `rel="next"`, so no client can tell a full page from the last page
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 6
- Avg merge
- 1h 25m
- Merged PRs (30d)
- 3
Description
## Summary
Nothing in a paged response says whether more results exist. A client has to infer it from the number of records it received, and both available inferences are wrong:
- Stopping when `results.length < limit` **silently drops records**, because a clamped `limit` produces a short page that is not a final page.
- Stopping when `results.length === 0` and advancing by `results.length` **never terminates**, because past the `skip` maximum every request returns the same page.
The fix is to stop making clients guess. Emit `Link: <…>; rel="next"` (RFC 8288) while another page exists and omit it when the result set is exhausted. Clients follow `next` and stop when it is absent, which makes every length-based stop condition irrelevant in one move.
This is the centerpiece of the paging work. It is the only item that requires per-controller changes rather than a change to the shared `getPagination()` helper.
## Why this matters
**Both client shapes fail today, in opposite directions.** From the parent draft, both run against the same query on the same deployment:
```text
# stop on short page, limit=1000
req #1: skip=0 -> 500 docs
=> 500 records (client believes the walk is complete; the query matches more than 100000)
# stop on empty page, limit=100, entered at skip=99800
req #3: skip=100000 -> 100 docs, first=4c0ef019
req #4: skip=100100 -> 100 docs, first=4c0ef019
req #5: skip=100200 -> 100 docs, first=4c0ef019
...never terminates
```
With `rel="next"` the first client gets a `next` link on its 500-record page and keeps going; the second gets no `next` link at the end and stops. Neither has to know what the server's maximums are.
**A client that wants to defend itself currently cannot.** No response header carries the applied `limit`/`skip` or the maximums. The only `Link` emitted is the JSON-LD context (`utils.js:163`). `HEAD /query` returns the `Content-Length` of the page it would have sent, not a total count, and answers an empty page with 404 where `POST` answers `200 []`, so it cannot be used to probe ahead. `GET /v1/api` lists endpoint descriptions and nothing else. The OpenAPI contract does not mention `limit` or `skip` at all.
**We documented the problem instead of fixing it.** `public/API.html` warns three times (lines 507, 612, 730) that "your application may experience strange behavior with large limits, such as ?limit=1000". The strange behavior is the silent truncation. Nothing enforces the warning, and a client that sets its own page size above 500 fails silently from that point on.
## Affected lines
| File | Line | Current |
|------|------|---------|
| `utils.js` | 163 | `configureLDHeadersFor()` assigns `Link` as a single string — the context link only |
| `controllers/crud.js` | 87 | `/query` — `.limit(limit)`, needs `limit + 1` |
| `controllers/history.js` | 91 | `HEAD /query` — same |
| `controllers/search.js` | 211, 216 | `$limit: limit + skip` per branch, needs `+ 1` |
| `controllers/search.js` | 282, 368 | `merged.slice(skip, skip + limit)` — trim point for `/search`, `/search/phrase` |
| `controllers/search.js` | 446, 540, 672 | Same shape in the unmounted `searchFuzzily`, `searchWildly`, `searchAlikes` |
| `controllers/gog.js` | 36, 167 | `/gog/*InManuscript`, default 50 |
## Proposed change
### Know whether another page exists
Over-fetch one record and trim it before serialization. The extra record is never returned; its existence is the signal.
- `/query` and `HEAD /query`: `db.find(props).sort({_id:1}).limit(limit + 1).skip(skip)`
- `/search` and friends: `$limit: limit + skip + 1` per branch, then slice as today
- `/gog/*InManuscript`: the equivalent in each aggregation
### Emit the header
```text
Link: ; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json",
; rel="next"
```
Points worth getting right:
- **Append, do not replace.** `configureLDHeadersFor()` (`utils.js:163`) currently assigns `Link` as a single string. The context link must survive.
- **Absolute URLs from `process.env.RERUM_PREFIX`**, which is already the deployment's base (`https://store.rerum.io/v1`). Do not build them from the request host.
- **`next` is absent on the final page.** Absence is the termination signal, so it has to be genuinely absent, not an empty value.
- **`rel="prev"` and `rel="first"`** are cheap to add and make the header a complete RFC 8288 set. `rel="last"` requires a total count and should not be attempted.
- **`/query` and `/search` are POST endpoints.** The `next` URL carries the pagination parameters; the client re-sends the same request body to it. This is the ordinary pattern for POST-based search APIs, but it needs saying explicitly in the documentation or clients will expect a GET.
- `Access-Control-Expose-Headers` is already `*` (`app.js:52`), so browser clients can read the header without further CORS work.
### Fix the published example
Revise `pagedQuery` at `public/API.html:555` to follow `rel="next"`, and replace the three "strange behavior" warnings with the actual maximums and the actual mechanism.
## Notes
- **Additive and non-breaking.** Existing clients ignore the header and behave exactly as they do now. This is the one item in the set that can ship without client coordination, which is an argument for landing it before #301 so that the `skip` rejection has somewhere to send people.
- Depends on #300. A `next` link over a non-deterministic order promises more than the server can keep.
- For `/search`, `rel="next"` is honest about page boundaries but the pool it pages over is still missing documents until #307 lands. Both are in the minimum set for a client to walk `/search` completely.
- Once #303 lands, the `/query` `next` URL should carry a cursor instead of a `skip`. Clients that follow the link get unbounded depth for free, without changing their code. Design the header so that swap is invisible to a conforming client.
- The `Link` response header needs to be declared in `openapi/contracts/core-provider.openapi.yaml`; see #305.
- #252 is still open and its third recommendation is cursor-based pagination, which this issue is the first half of.
## Acceptance criteria
- [ ] Paged responses carry `Link: …; rel="next"` while more results exist, and omit it on the final page
- [ ] The JSON-LD context link is still present alongside `rel="next"`
- [ ] The over-fetched record is never returned to the client, at any `limit` including the maximum
- [ ] `/query`, `HEAD /query`, `/search`, `/search/phrase`, and both `/gog/*InManuscript` endpoints all emit it
- [ ] The unmounted `searchFuzzily`, `searchWildly`, and `searchAlikes` are updated to match, so the pattern is not carried forward when they are routed
- [ ] A client following only `rel="next"` walks a result set to completion and terminates, with no length-based stop condition
- [ ] Regression tests cover `rel="next"` presence, its absence on the final page, and the untrimmed extra record never appearing
- [ ] `public/API.html` publishes a `pagedQuery` example that follows `rel="next"`
Contributor guide
Assessment
This issue has not been assessed yet.