Store git commit metadata (author, parents) for hashes we already ingest
- Dominant language
- Python
- Stars
- 9
- Forks
- 31
- Avg merge
- 3d 19h
- Merged PRs (30d)
- 16
Description
## Context
`checkouts` stores CI submissions, not git objects. For each checkout we currently keep:
- `git_commit_hash` / `git_commit_name` / `git_commit_tags`
- `git_commit_message` (often empty; submitters rarely send it)
- `git_repository_url` / `git_repository_branch`
KCIDB has no author, committer, or parent fields, so we cannot get this from ingestion. We also have no parent chain, which is why [#1958](https://github.com/kernelci/dashboard/issues/1958) / [#2054](https://github.com/kernelci/dashboard/pull/2054) had to approximate “next checkout” with `start_time`.
The same `git_commit_hash` can appear on many checkouts (origins, labs). Author / parents are properties of the git object, not of a checkout row. Storing them only on `checkouts` would duplicate data and go stale independently per origin.
## Goal
Persist git metadata for commits we already see in `checkouts`, **plus their ancestors**, filled from the real repositories (not from KCIDB submissions). Ancestor rows are required so first-parent walks stay correct when CI skipped a commit.
### Store at least
- author name, email, date
- committer name, email, date
- parent hashes (ordered; first parent = mainline parent on merges) as rows in `commit_parents`
- subject (first line of the message)
- full message on **`commits`**, not on `checkouts`
Do **not** backfill `checkouts.git_commit_message`. Later: rewrite readers to `commits.message`, copy unreachable submitter messages onto `commits`, then drop the checkout column.
Nice to have later: `trees` / named refs, merge as `ord > 0`, untested-branch analysis. A commit is not owned by a single tree — membership is a walk from branch tips.
## Proposed shape
- `commits`: surrogate `id` PK, unique `git_commit_hash`, author/committer/subject/message. Join from `checkouts.git_commit_hash` for now (`commit_id` FKs later).
- `commit_parents`: `commit_id`, `parent_id`, `ord` (`0` = first parent).
Do **not** add these columns onto `checkouts`. Do **not** store `fetched_from_url` on the commit (URL is checkout/tree info; provenance belongs on the sync run if anywhere).
## Filling it
Background command / cron on a **persistent treeless** repo (many remotes, `--filter=tree:0`):
1. Fetch refs from an allowlist of known-good tree URLs (treeproof-style), not from arbitrary `checkouts.git_repository_url`.
2. Enumerate new commit objects (`rev-list` new tips `--not` previously ingested tips, or equivalent).
3. Parse and upsert `commits` + `commit_parents` in topological order.
4. Skip / log per-remote failures. Job must not die on one tree.
Must tolerate missing objects, private remotes, and the malformed `git_repository_url` cases from [#1952](https://github.com/kernelci/dashboard/issues/1952) / [#1953](https://github.com/kernelci/dashboard/issues/1953) without crashing the job.
Checkout-hash-only backfill ([#2091](https://github.com/kernelci/dashboard/issues/2091), closed) cannot fill gaps CI skipped. One-shot SHA fetch ([#2090](https://github.com/kernelci/dashboard/issues/2090)) is the fallback for hashes the mirror does not cover, not the fill loop.
Queries use Postgres, not live git.
## Why this is useful
- Real first-parent history instead of ordering checkouts by `start_time`
- Author on tree / commit / issue views
- Search / filter by author
- Honest merge vs first-parent walks
- Later: skipped commits / untested branches
## Out of scope
- Changing the KCIDB submission schema or waiting for origins to send this
- UI (follow-up once the table is populated)
- Rewriting treeproof / tree-name mapping ([#1953](https://github.com/kernelci/dashboard/issues/1953))
- Querying git from request handlers
## Sub-issues
- [#2089](https://github.com/kernelci/dashboard/issues/2089) schema (`commits` + `commit_parents`)
- [#2090](https://github.com/kernelci/dashboard/issues/2090) parse + one-shot SHA fetch (no DB)
- [#2109](https://github.com/kernelci/dashboard/issues/2109) sync job (replaces #2091)
- [#2092](https://github.com/kernelci/dashboard/issues/2092) first-parent next checkout
## Related
- [#1958](https://github.com/kernelci/dashboard/issues/1958) next checkout after last seen
- [#1509](https://github.com/kernelci/dashboard/issues/1509) reject checkouts without `commit_hash`
Contributor guide
Research direction
Start with sub-issues #2089, #2090, and #2109 to trace the proposed schema, git parsing, and background sync entry points. Confirm how a persistent treeless repository, allowlisted remotes, and Postgres upserts should populate commits and ordered commit_parents while tolerating failures; done means ancestor-aware metadata is stored without changing checkouts or crashing on bad remotes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, postgresql, python
- Domain
- backend, databases, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100