Columnar storage / compression
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 159
- Forks
- 156
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 16
Description
The commits table of the mergebot is already getting pretty large (2.5GB). This is almost entirely from the statuses column, which does not compress that well intrinsically but is extremely redundant between rows, although not identical (because various items include build numbers or identifiers, commit authors, runtime duration, ...)
There are, most likely, three good options here:
- prune older commits, commits are fundamentally just a vessel to move statuses over to PRs and stagings, once those are closed, merged, or otherwise completed the commits stop being useful (eventually), this requires no external dependencies or extras, and can even be done on a point basis (e.g. every few months clean all the commits older than 6 months which don't match any open PR, or something), even on commits which are conserved (staging heads and commits) it should be possible to remove the statuses themselves
- encode runbot patterns and perform manual content extraction into known templates
- partitioning based on the commit's write date, although that doesn't really save on retrieving backups / duplicating the DB I think, unless the partition can use a different storage form than the source?
- use some sort of compressible columnar storage e.g. timescaledb, pg-xpatch (no support for UPDATE tho), ...
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the mergebot commits table and its statuses column, then compare the four storage approaches listed in the issue, including pruning and PostgreSQL extensions. No files, tests, or entry points are identified, and the work is not done until a storage strategy is selected and its impact on retained commit data and backups is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100