chaoss / chaoss/CollectOSS

Prevent duplicate commit insertion on facade re-processing

Open
#260 3 comments 0 reactions 0 assignees View on GitHub
task:facade theoretical
Dominant language
Python
Stars
13
Forks
17
Avg merge
6h 59m
Merged PRs (30d)
1

Description

> [!NOTE]
> Migrated from [augurlabs/augur#3790](https://github.com/augurlabs/augur/issues/3790)
> Originally opened by `@MoralCode` on 2026-03-25

---

the `commits` table has a primary key that is derived only from a sequence (i.e. an autoincrementing integer)

When facade `analyze_commits_in_parallel` runs (to insert all new commit change information into the commits table), it does not utilize Augur's existing upsert (`bulk_insert_dicts` or similar) logic. Instead it simply inserts always, causing new IDs to be generated and new rows to be added.

If the run of analyzing commits is a rerun (i.e. the repo previously was fully collected, but the admin reset the last collection date to force recollection, meaning many of the commits are already in the table), this will simply generate duplicate rows, contributing to the size growth of one of the largest tables in Augur.

In order to use upserts for the commits table, we need a compound primary key based on the actual data. Given this table is actually more accurately described as commit_changes (#211), i propose this constraint `UniqueConstraint("repo_id", "cmt_commit_hash", "cmt_filename", name="commit-changes-unique"),`.

Contributor guide

Open the contributing guide

Research direction

Start with the facade's analyze_commits_in_parallel entry point and compare its insertion path with the existing bulk_insert_dicts upsert logic. Inspect the commits table schema and determine how the proposed repo_id, cmt_commit_hash, and cmt_filename uniqueness constraint fits the storage layer. Done means rerunning commit analysis does not create duplicate rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.