CenterForDigitalHumanities / CenterForDigitalHumanities/rerum_server_nodejs

Add cursor-based continuation to `/query` so paging depth is unbounded and cost is flat

Open
#303 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
6
Avg merge
1h 25m
Merged PRs (30d)
3

Description

## Summary

`skip` cannot page past the configured maximum. Once #301 lands, a request past it is a 400 instead of a repeating page — honest, but it also means a client is told it may not read past 100000 records. That is a correct contract and a worse one than clients think they have today.

Keyset (cursor) continuation removes the ceiling. `/query` already has a unique, indexed sort key once #300 lands, so a cursor is just "resume after this `_id`":

```js
db.find({ ...props, _id: { $gt: lastId } }).sort({ _id: 1 }).limit(limit + 1)
```

Cost is flat with depth, because Mongo seeks the index rather than counting past `skip` documents. Depth is unbounded, because there is no offset to cap.

`skip` stays. **Decision recorded on the parent thread:** keep both modes and document them — `skip` as the bounded, random-access mode, cursors as the unbounded, sequential one.

## Why this matters

**It is what makes the `skip` rejection defensible.** Rejecting `skip > 100000` with no alternative narrows what the API can do. Rejecting it while handing back a `rel="next"` link that pages indefinitely is a straight improvement. These two issues are better shipped close together than far apart.

**Deep `skip` is expensive even inside the cap.** Mongo walks and discards every skipped document. The parent draft measured `/query` at 340-445 ms across `skip=0` through `skip=99000`, so this is not currently a crisis on `/query` — but it is linear work that a cursor makes constant, and it is the shape that matters once collections grow.

**Clients cannot build a cursor themselves, and should not.** `idNegotiation()` deletes `_id` from every response body (`controllers/utils.js:70`) and only reattaches it as `id` when the `@context` matches a known mapping. So the server must mint the token and hand it back. That is the right design anyway: an opaque, server-minted token keeps the encoding an implementation detail, and a client that only follows `rel="next"` needs no code change when the encoding changes.

## Proposed change

### The token

Add a `cursor` query parameter to `/query`. It is opaque to clients and appears only in the server's `rel="next"` link:

```text
Link: ; rel="next"
```

The token encodes the last `_id` of the page just served. Base64url of a small JSON envelope is sufficient. Two requirements:

- **Encode the BSON type, not just the value.** `newID()` returns `new ObjectId().toHexString()`, so modern RERUM `_id` values are **strings**. Legacy v0 documents on the dev collection carry embedded-object `_id` values (roughly 6400; production has none), and some legacy documents may carry true `ObjectId` values. BSON type ordering gives a total order across all three, so `$gt` paging is correct — but a token that round-trips a bare hex string will compare against the wrong type and silently skip or repeat a block of documents at each type boundary. This is the detail most likely to be gotten wrong, and it will only fail on dev, which is where clients are developed.
- **Validate and reject a malformed token with 400.** Same principle as the rest of #301: do not guess.

### Interaction with `skip`

- `cursor` and `skip` together is a 400. They are two different positioning schemes and combining them has no coherent meaning.
- `limit` applies to both modes and is clamped identically.
- `rel="next"` on `/query` should carry a `cursor`, not a `skip`, once this lands. A client that already follows the link inherits unbounded depth with no change on its side. That transparency is the argument for shipping #302 first and this second.
- `skip` keeps working, keeps its maximum, and is documented as the random-access mode for jumping to a known offset within the cap.

### Scope

`/query` only. See the note below on `/search`.

## Notes

- **`/search` is deliberately excluded.** Under the current in-memory paging (`controllers/search.js:282`), a cursor could only encode an offset into the merged array. Same cost, same ceiling, dressed up as a cursor — worse than not shipping one, because it would advertise a guarantee the implementation does not provide. Revisit after #309, at which point Atlas Search's own `searchAfter` / `searchSequenceToken` paging is the genuine equivalent. **Availability on our cluster tier and server version needs confirming before that goes in any plan** — do not assume it.
- Depends on #300 (needs a sort key) and #302 (needs somewhere to put the token).
- A keyset cursor is stable under insert in a way `skip` is not: a document inserted before the cursor position does not shift the remaining pages. Given RERUM's mark-deleted-never-remove policy, this makes cursor walks meaningfully more correct than offset walks over a live collection, which is a second reason to prefer them beyond cost.
- This is #252's third recommendation. Worth linking there when filed.
- `HEAD /query` should accept `cursor` too, so both verbs stay interchangeable. See #304.

## Acceptance criteria

- [ ] `/query` accepts an opaque `cursor` parameter and returns the page following it
- [ ] The `rel="next"` link on `/query` carries a `cursor`, and a client following only that link pages past the `skip` maximum and terminates correctly
- [ ] Cursor paging returns every matching document exactly once across mixed `_id` types, verified on the dev collection where string, `ObjectId`, and embedded-object `_id` values coexist
- [ ] A cursor walk and a `skip` walk of the same query return the same records in the same order, within the range where `skip` is legal
- [ ] `/query` latency is flat with respect to depth under cursor paging, measured to the depth `skip` cannot reach
- [ ] A malformed or unparseable `cursor` returns 400
- [ ] `cursor` combined with `skip` returns 400
- [ ] `skip` continues to work within its maximum, and both modes are documented in `public/API.html` and the OpenAPI contract

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.