[#2079] Sync mirrored trees into commits
- Dominant language
- Python
- Stars
- 9
- Forks
- 31
- Avg merge
- 3d 19h
- Merged PRs (30d)
- 16
Description
## Parent
https://github.com/kernelci/dashboard/issues/2079
Schema: https://github.com/kernelci/dashboard/issues/2089
Parse / one-shot SHA fetch: https://github.com/kernelci/dashboard/issues/2090
Replaces [#2091](https://github.com/kernelci/dashboard/issues/2091) (checkout-hash backfill). Filling from distinct `checkouts.git_commit_hash` cannot produce a continuous first-parent chain when CI skips commits ([#2079](https://github.com/kernelci/dashboard/issues/2079) discussion).
Management command / cron that keeps `commits` and `commit_parents` filled with the **ancestor closure** of commits we care about: every ancestor of hashes we see in `checkouts` on trees we mirror. Postgres is the query store. Git is an ingest cache.
## In scope
- Persistent **treeless** repo (`--filter=tree:0`), many remotes. Do not throw it away per hash. Do not write under `BACKEND_VOLUME_DIR`; dedicated volume / path, passed into git.
- Remotes from a tree allowlist (treeproof-style known-good URLs), **not** from arbitrary `checkouts.git_repository_url`.
- Cron: `git fetch` remotes; enumerate **new** objects (`rev-list` new tips `--not` previously ingested tips, or equivalent); parse via the helper in [#2090](https://github.com/kernelci/dashboard/issues/2090) (existing-repo path, no per-hash fetch).
- Upsert `commits` then `commit_parents` in **topological order** so parent rows exist before edges. No stub rows.
- Continue on per-remote failure (bad URL, missing object, full pack). Log and move on. Job must not die on one tree.
- Dry-run: fetch/parse optional, **write nothing**.
- Do **not** write `checkouts.git_commit_message`. Message lives on `commits`.
Optional safety, not the main loop: report `checkouts.git_commit_hash` still missing from `commits` (and optionally call the #2090 SHA helper for those gaps).
## Out of scope
- `trees` / `named_refs` tables (follow-up).
- Snapshot policy for `update_db` (follow-up once we know row counts).
- Rewriting readers of `checkouts.git_commit_message` / dropping that column (follow-up; copy unreachable messages onto `commits` first).
## Tests
Mocked parse/fetch: upserts commits + first-parent edges in topo order. One remote fails, others still write. Dry-run writes nothing. Skipped intermediate commit between two checkouts still lands in `commits`.
Contributor guide
Research direction
Start with parent issue #2079, schema issue #2089, and the parse/one-shot SHA helper in #2090, then trace the management-command or cron entry point. Verify the existing-repository Git flow, topological upserts into commits and commit_parents, and per-remote failure handling. Done means the listed mocked fetch/parse, skipped-commit, failure-continuation, and dry-run tests pass without writing in dry-run mode.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, postgresql, python
- Domain
- backend, databases, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100