Automattic / Automattic/mongoose

Query.prototype.after() keyset pagination boundaries

Closed
#16,410 18 comments 0 reactions 0 assignees View on GitHub
enhancement new feature
Dominant language
JavaScript
Stars
27.5k
Forks
4k
Avg merge
2d 7h
Merged PRs (30d)
35

Description

### Prerequisites

- [x] I have written a descriptive issue title
- [x] I have searched existing issues to ensure the feature has not already been requested

### 🚀 Feature Proposal

Add `Query.prototype.after(boundary)`, which derives a keyset (seek) boundary predicate from the query's existing `sort()` specification and merges it into the query filter.

```js
const posts = await Post.find({ authorId })
.sort({ createdAt: -1, _id: -1 })
.after(lastDocOfPreviousPage)
.limit(20)
.lean();
```

### Motivation

Mongoose already ships pagination: `skip()` and `limit()` are in core. They are also the wrong tool for most collections, because `skip(n)` requires the server to walk and discard `n` documents, so latency grows linearly with page depth. Core currently blesses the O(n) approach and leaves the O(log n) approach entirely to userland.

The concept of keyset pagination is easy. The part people get wrong is the boundary predicate, and Mongoose is uniquely positioned to get it right because it owns the schema.

**1. The tie-breaker expansion.** MongoDB has no row-value comparison, so a multi-key boundary must be expanded lexicographically by hand:

```js
$or: [
{ createdAt: { $lt: t } },
{ createdAt: t, _id: { $lt: id } },
]
```

Written incorrectly — most commonly as `{ createdAt: { $lt: t } }` alone — the query silently drops or duplicates documents whenever timestamps collide. There is no error, no warning, and no failing test unless someone thought to seed colliding values. It surfaces months later as a support ticket.

**2. Casting.** Boundary values arrive from an HTTP client as strings. Every hand-rolled implementation contains manual `new Date(...)` and `new Types.ObjectId(...)` calls, applied by hand, per sort key. Mongoose already knows each path's `SchemaType` and already has cast machinery. Userland plugins cannot reuse it cleanly; core can, for free.

**3. Filter merging.** Assigning `filter.$or` clobbers any `$or` the caller already had. The correct merge is `$and`-wrapping. This is a second silent-wrong-results bug, and it is invisible until someone adds an unrelated `$or` to a filter six months later.

All three are mechanical, schema-aware, and exactly the class of problem a library should own.

### Example

## Proposed API

```
Query.prototype.after(boundary)
```

**`boundary`** as one of:

- a hydrated document
- a lean/POJO result (e.g. the last element of the previous page)
- a plain object of boundary values, e.g. `{ createdAt: '2026-01-14T09:00:00Z', _id: '507f1f77bcf86cd799439011' }`

Values are cast through the schema paths named in the sort, so raw strings from a query parameter work without caller-side conversion. Dotted sort paths (`'author.name'`) are read from the boundary via the same path resolution used elsewhere in Mongoose.

Chain order is irrelevant; `.after()` may be called before or after `.sort()`. The predicate is materialised at cast/exec time, not at call time.

### Semantics

Given sort keys `k₁…kₙ` with directions `d₁…dₙ` and boundary values `v₁…vₙ`, `after()` produces:

```js
{ $or: [
{ k₁: { [op(d₁)]: v₁ } },
{ k₁: v₁, k₂: { [op(d₂)]: v₂ } },
{ k₁: v₁, k₂: v₂, k₃: { [op(d₃)]: v₃ } },

] }
```

where `op(1) === '$gt'` and `op(-1) === '$lt'`. The result is merged into the existing filter under `$and` when the filter already contains `$or`, and assigned directly otherwise.

### Validation rules

`after()` throws (rather than producing a subtly wrong query) when:

| Condition | Rationale |
| --- | --- |
| No `sort()` on the query | A boundary is meaningless without an ordering. |
| A sort key is missing from `boundary` | Treating a missing key as `null` is not the same comparison and would silently shift the page. `undefined` must not be coerced. |
| `skip()` is also set | Combining seek and offset is almost certainly a bug. |
| The sort is not provably total | See below. |

**Totality.** Keyset pagination is only correct if the sort uniquely orders every document. `after()` accepts the sort if its final key is `_id`, or if that key is declared `unique: true` in the schema which is information Mongoose already has, otherwise it throws with a message naming the fix (append `_id` to the sort). An escape hatch, `after(boundary, { allowNonUniqueSort: true })`, exists for callers who know something the schema doesn't.

Deliberately **not** auto-appending `_id`: that would silently change the caller's sort, and therefore change which index the query needs.

**Index hinting (opt-in).** Mongoose can already enumerate declared indexes via `schema.indexes()`. Under an opt-in flag such as `mongoose.set('warnUnindexedKeyset', true)`, `after()` emits a one-time warning per model+sort shape when no declared index has the sort keys as a prefix (in matching or fully reversed direction). This is the other silent failure mode, a correct query that collapses to a COLLSCAN and is cheap to detect. Warning only, never throwing; runtime indexes may exist that the schema doesn't declare.

## Examples

**Simple `_id` feed**

```js
Message.find({ roomId }).sort({ _id: -1 }).after({ _id: req.query.cursor }).limit(50);
```

**Compound sort with string boundary values**

```js
Order.find({ status: 'open' })
.sort({ placedAt: -1, _id: -1 })
.after({ placedAt: req.query.t, _id: req.query.id }) // both cast by schema
.limit(25)
.lean();
```

**Filter that already contains `$or`**

```js
Post.find({ $or: [{ visibility: 'public' }, { authorId: me }] })
.sort({ score: -1, _id: -1 })
.after(last)
.limit(20);
// → { $and: [ { $or: [...caller's] }, { $or: [...boundary] } ] }
```

Contributor guide

Open the contributing guide

Research direction

The proposed entry point is Query.prototype.after(); begin by tracing existing sort(), filter merging, and cast/exec behavior. Review schema path casting and schema.indexes() for the stated schema-aware behavior. Done means keyset predicates handle compound sorts, preserve existing filters, validate the listed conditions, and support the documented boundary forms.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, mongodb
Domain
backend, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.