finos / finos/git-proxy

Reduce stored push document size and rollup scan cost across sink backends

Open
#1,691 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
249
Forks
176
Avg merge
3d 8h
Merged PRs (30d)
20

Description

Discussion issue, applies to every sink backend (fs, mongo, and the PostgreSQL sink proposed in #1687).

Every push row stores the complete diff twice. `getDiff` writes it to both `steps[].logs` (via `step.log(diff)`) and `steps[].content` (via `step.setContent(diff)`), and `writeAudit` persists the whole action document as-is in all backends. Large pushes therefore produce multi-megabyte rows, and everything that touches push documents pays for it:

- The repo activity rollup scans all push rows on every repos page. Indexing can make the scan cheap on PostgreSQL, but it remains O(number of pushes) on every backend, and on mongo/fs it reads the full documents.
- List endpoints and exports ship or filter over fat documents.
- Storage grows with diff size rather than with metadata size.

Ideas worth weighing, none of which are backend-specific:

1. Store the diff once, not twice: drop the duplicate from `steps[].logs` or make `setContent` and the log reference one value.
2. Move diff content out of the push document into its own store or table, loaded only by the push-detail view.
3. A retention or truncation policy for step logs and diff content on old, settled pushes.
4. Maintain the repo activity rollup incrementally (a counters document or table updated on push transitions) instead of recomputing it by scanning all pushes.

Filing as a design discussion before any implementation; the immediate, index-level mitigations for the PostgreSQL sink are tracked separately.

Contributor guide

Open the contributing guide

Research direction

Start by tracing getDiff and writeAudit, then compare how push documents are stored and scanned across the fs, mongo, and proposed PostgreSQL sinks in #1687. Evaluate the listed storage, retention, and rollup options without assuming a backend-specific fix. Done means selecting and documenting a backend-independent design with clear effects on document size and repo activity scans.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, postgresql, typescript
Domain
backend, databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.