kernelci / kernelci/dashboard

[#2079] Sync mirrored trees into commits

Open
#2,109 0 comments 0 reactions 0 assignees View on GitHub
Backend Data Quality Database enhancement
Dominant language
Python
Stars
9
Forks
31
Avg merge
3d 19h
Merged PRs (30d)
16

Description

## Parent

https://github.com/kernelci/dashboard/issues/2079
Schema: https://github.com/kernelci/dashboard/issues/2089
Parse / one-shot SHA fetch: https://github.com/kernelci/dashboard/issues/2090

Replaces [#2091](https://github.com/kernelci/dashboard/issues/2091) (checkout-hash backfill). Filling from distinct `checkouts.git_commit_hash` cannot produce a continuous first-parent chain when CI skips commits ([#2079](https://github.com/kernelci/dashboard/issues/2079) discussion).

Management command / cron that keeps `commits` and `commit_parents` filled with the **ancestor closure** of commits we care about: every ancestor of hashes we see in `checkouts` on trees we mirror. Postgres is the query store. Git is an ingest cache.

## In scope

- Persistent **treeless** repo (`--filter=tree:0`), many remotes. Do not throw it away per hash. Do not write under `BACKEND_VOLUME_DIR`; dedicated volume / path, passed into git.
- Remotes from a tree allowlist (treeproof-style known-good URLs), **not** from arbitrary `checkouts.git_repository_url`.
- Cron: `git fetch` remotes; enumerate **new** objects (`rev-list` new tips `--not` previously ingested tips, or equivalent); parse via the helper in [#2090](https://github.com/kernelci/dashboard/issues/2090) (existing-repo path, no per-hash fetch).
- Upsert `commits` then `commit_parents` in **topological order** so parent rows exist before edges. No stub rows.
- Continue on per-remote failure (bad URL, missing object, full pack). Log and move on. Job must not die on one tree.
- Dry-run: fetch/parse optional, **write nothing**.
- Do **not** write `checkouts.git_commit_message`. Message lives on `commits`.

Optional safety, not the main loop: report `checkouts.git_commit_hash` still missing from `commits` (and optionally call the #2090 SHA helper for those gaps).

## Out of scope

- `trees` / `named_refs` tables (follow-up).
- Snapshot policy for `update_db` (follow-up once we know row counts).
- Rewriting readers of `checkouts.git_commit_message` / dropping that column (follow-up; copy unreachable messages onto `commits` first).

## Tests

Mocked parse/fetch: upserts commits + first-parent edges in topo order. One remote fails, others still write. Dry-run writes nothing. Skipped intermediate commit between two checkouts still lands in `commits`.

Contributor guide

Open the contributing guide

Research direction

Start with parent issue #2079, schema issue #2089, and the parse/one-shot SHA helper in #2090, then trace the management-command or cron entry point. Verify the existing-repository Git flow, topological upserts into commits and commit_parents, and per-remote failure handling. Done means the listed mocked fetch/parse, skipped-commit, failure-continuation, and dry-run tests pass without writing in dry-run mode.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, postgresql, python
Domain
backend, databases, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.