Reduce stored push document size and rollup scan cost across sink backends
- Dominant language
- TypeScript
- Stars
- 249
- Forks
- 176
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 20
Description
Discussion issue, applies to every sink backend (fs, mongo, and the PostgreSQL sink proposed in #1687).
Every push row stores the complete diff twice. `getDiff` writes it to both `steps[].logs` (via `step.log(diff)`) and `steps[].content` (via `step.setContent(diff)`), and `writeAudit` persists the whole action document as-is in all backends. Large pushes therefore produce multi-megabyte rows, and everything that touches push documents pays for it:
- The repo activity rollup scans all push rows on every repos page. Indexing can make the scan cheap on PostgreSQL, but it remains O(number of pushes) on every backend, and on mongo/fs it reads the full documents.
- List endpoints and exports ship or filter over fat documents.
- Storage grows with diff size rather than with metadata size.
Ideas worth weighing, none of which are backend-specific:
1. Store the diff once, not twice: drop the duplicate from `steps[].logs` or make `setContent` and the log reference one value.
2. Move diff content out of the push document into its own store or table, loaded only by the push-detail view.
3. A retention or truncation policy for step logs and diff content on old, settled pushes.
4. Maintain the repo activity rollup incrementally (a counters document or table updated on push transitions) instead of recomputing it by scanning all pushes.
Filing as a design discussion before any implementation; the immediate, index-level mitigations for the PostgreSQL sink are tracked separately.
Contributor guide
Research direction
Start by tracing getDiff and writeAudit, then compare how push documents are stored and scanned across the fs, mongo, and proposed PostgreSQL sinks in #1687. Evaluate the listed storage, retention, and rollup options without assuming a backend-specific fix. Done means selecting and documenting a backend-independent design with clear effects on document size and repo activity scans.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb, postgresql, typescript
- Domain
- backend, databases, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100