kernelci / kernelci/dashboard

Store git commit metadata (author, parents) for hashes we already ingest

Open
#2,079 10 comments 1 reaction 0 assignees View on GitHub
Backend Data Quality Database enhancement
Dominant language
Python
Stars
9
Forks
31
Avg merge
3d 19h
Merged PRs (30d)
16

Description

## Context

`checkouts` stores CI submissions, not git objects. For each checkout we currently keep:

- `git_commit_hash` / `git_commit_name` / `git_commit_tags`
- `git_commit_message` (often empty; submitters rarely send it)
- `git_repository_url` / `git_repository_branch`

KCIDB has no author, committer, or parent fields, so we cannot get this from ingestion. We also have no parent chain, which is why [#1958](https://github.com/kernelci/dashboard/issues/1958) / [#2054](https://github.com/kernelci/dashboard/pull/2054) had to approximate “next checkout” with `start_time`.

The same `git_commit_hash` can appear on many checkouts (origins, labs). Author / parents are properties of the git object, not of a checkout row. Storing them only on `checkouts` would duplicate data and go stale independently per origin.

## Goal

Persist git metadata for commits we already see in `checkouts`, **plus their ancestors**, filled from the real repositories (not from KCIDB submissions). Ancestor rows are required so first-parent walks stay correct when CI skipped a commit.

### Store at least

- author name, email, date
- committer name, email, date
- parent hashes (ordered; first parent = mainline parent on merges) as rows in `commit_parents`
- subject (first line of the message)
- full message on **`commits`**, not on `checkouts`

Do **not** backfill `checkouts.git_commit_message`. Later: rewrite readers to `commits.message`, copy unreachable submitter messages onto `commits`, then drop the checkout column.

Nice to have later: `trees` / named refs, merge as `ord > 0`, untested-branch analysis. A commit is not owned by a single tree — membership is a walk from branch tips.

## Proposed shape

- `commits`: surrogate `id` PK, unique `git_commit_hash`, author/committer/subject/message. Join from `checkouts.git_commit_hash` for now (`commit_id` FKs later).
- `commit_parents`: `commit_id`, `parent_id`, `ord` (`0` = first parent).

Do **not** add these columns onto `checkouts`. Do **not** store `fetched_from_url` on the commit (URL is checkout/tree info; provenance belongs on the sync run if anywhere).

## Filling it

Background command / cron on a **persistent treeless** repo (many remotes, `--filter=tree:0`):

1. Fetch refs from an allowlist of known-good tree URLs (treeproof-style), not from arbitrary `checkouts.git_repository_url`.
2. Enumerate new commit objects (`rev-list` new tips `--not` previously ingested tips, or equivalent).
3. Parse and upsert `commits` + `commit_parents` in topological order.
4. Skip / log per-remote failures. Job must not die on one tree.

Must tolerate missing objects, private remotes, and the malformed `git_repository_url` cases from [#1952](https://github.com/kernelci/dashboard/issues/1952) / [#1953](https://github.com/kernelci/dashboard/issues/1953) without crashing the job.

Checkout-hash-only backfill ([#2091](https://github.com/kernelci/dashboard/issues/2091), closed) cannot fill gaps CI skipped. One-shot SHA fetch ([#2090](https://github.com/kernelci/dashboard/issues/2090)) is the fallback for hashes the mirror does not cover, not the fill loop.

Queries use Postgres, not live git.

## Why this is useful

- Real first-parent history instead of ordering checkouts by `start_time`
- Author on tree / commit / issue views
- Search / filter by author
- Honest merge vs first-parent walks
- Later: skipped commits / untested branches

## Out of scope

- Changing the KCIDB submission schema or waiting for origins to send this
- UI (follow-up once the table is populated)
- Rewriting treeproof / tree-name mapping ([#1953](https://github.com/kernelci/dashboard/issues/1953))
- Querying git from request handlers

## Sub-issues

- [#2089](https://github.com/kernelci/dashboard/issues/2089) schema (`commits` + `commit_parents`)
- [#2090](https://github.com/kernelci/dashboard/issues/2090) parse + one-shot SHA fetch (no DB)
- [#2109](https://github.com/kernelci/dashboard/issues/2109) sync job (replaces #2091)
- [#2092](https://github.com/kernelci/dashboard/issues/2092) first-parent next checkout

## Related

- [#1958](https://github.com/kernelci/dashboard/issues/1958) next checkout after last seen
- [#1509](https://github.com/kernelci/dashboard/issues/1509) reject checkouts without `commit_hash`

Contributor guide

Open the contributing guide

Research direction

Start with sub-issues #2089, #2090, and #2109 to trace the proposed schema, git parsing, and background sync entry points. Confirm how a persistent treeless repository, allowlisted remotes, and Postgres upserts should populate commits and ordered commit_parents while tolerating failures; done means ancestor-aware metadata is stored without changing checkouts or crashing on bad remotes.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, postgresql, python
Domain
backend, databases, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.