CenterForDigitalHumanities / CenterForDigitalHumanities/rerum_server_nodejs

Query and version endpoints load unbounded results into memory

Open
#252 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
3
Forks
6
Avg merge
1h 25m
Merged PRs (30d)
3

Description

## Summary

Several RERUM endpoints load all matching documents into memory at once with no pagination or limit:

1. **`queryHeadRequest`** — `db.find(props).toArray()` with no limit. A broad query like `{"type": "Annotation"}` loads every matching document into memory before sending the response.

2. **`getAllVersions`** — Loads all version documents for an object at once. An object with 100+ versions loads all of them into a single array.

3. **`/history/{id}` and `/since/{id}`** — Both traverse the version chain and accumulate all results in memory.

## Evidence

From load testing (Run 4):

- **Phase 9 (DDoS), Query Flood scenario:** 200 VUs sending `POST /v1/api/query` with `{"type":"Annotation"}` for 3 minutes. Each request loaded ALL matching annotations into memory. RERUM survived but memory grew from 62MB to 276MB per worker.

- **Phase 9, History Tree Attack:** 50 VUs hitting `/v1/history/{id}` on an object with 100+ versions for 3 minutes. Each request loaded 100+ documents into memory simultaneously.

- **Phase 9, Query Amplification:** 100 VUs alternating between `{"type":"Annotation"}` and `{"label":{"$exists":true}}` — broad queries returning large result sets, stressing serialization and memory.

All scenarios contributed to the unbounded memory growth documented in #251.

## Recommendation

1. Add default pagination to query endpoints: `db.find(props).limit(100).toArray()` with a `limit` query parameter (default 100, max 1000)
2. Add a limit to `getAllVersions` — either cap at a reasonable number or stream results
3. Consider cursor-based pagination for large result sets to avoid loading everything at once

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.