odoo / odoo/runbot

Columnar storage / compression

Open
#1,383 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

mergebot
Dominant language
Python
Stars
159
Forks
156
Avg merge
1d 13h
Merged PRs (30d)
16

Description

The commits table of the mergebot is already getting pretty large (2.5GB). This is almost entirely from the statuses column, which does not compress that well intrinsically but is extremely redundant between rows, although not identical (because various items include build numbers or identifiers, commit authors, runtime duration, ...)

There are, most likely, three good options here:

  1. prune older commits, commits are fundamentally just a vessel to move statuses over to PRs and stagings, once those are closed, merged, or otherwise completed the commits stop being useful (eventually), this requires no external dependencies or extras, and can even be done on a point basis (e.g. every few months clean all the commits older than 6 months which don't match any open PR, or something), even on commits which are conserved (staging heads and commits) it should be possible to remove the statuses themselves
  2. encode runbot patterns and perform manual content extraction into known templates
  3. partitioning based on the commit's write date, although that doesn't really save on retrieving backups / duplicating the DB I think, unless the partition can use a different storage form than the source?
  4. use some sort of compressible columnar storage e.g. timescaledb, pg-xpatch (no support for UPDATE tho), ...

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the mergebot commits table and its statuses column, then compare the four storage approaches listed in the issue, including pruning and PostgreSQL extensions. No files, tests, or entry points are identified, and the work is not done until a storage strategy is selected and its impact on retained commit data and backups is established.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.