codegouvfr / codegouvfr/catalogi
Surface per-source data provenance on software details and unify user edits as a source
- Dominant language
- TypeScript
- Stars
- 43
- Forks
- 14
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 3
Description
## Problem
When viewing a software's detail page, data from all external sources (HAL, Wikidata, GitHub, GitLab, Comptoir du Libre, Zenodo, CNLL) is silently merged into a single flat object. The UI presents this merged result with no indication of which source contributed which field. Manually-entered data in the `softwares` table takes precedence over all external sources via `?? extData?.x` fallback chains, so editor input silently erases what every other source has to say. Users have no way to see "what does Wikidata think the URL is, vs. what an editor pinned?".
This issue redesigns the data model and the detail page UI so that:
- Every field's provenance is visible.
- Manual user input is treated as just another source with a configurable priority, not a special override layer.
- Each source's full contribution can be inspected in a dedicated UI.
## Goals
- Detail endpoint exposes the raw per-source data alongside the merged view.
- Users can open a provenance drawer to see, source-by-source, what every contributing source has to say about a software.
- Editors continue to see and edit "their" pinned values, but blank fields fall through to external sources rather than being silently saved as overrides.
- Reuser deployments that don't accept manual editing are unaffected — no `user_input` source row, no migrated rows, no UI change.
## Non-goals
- No edit history / audit log (deferred — `software_external_datas` rows remain "current state only").
- No per-field priority overrides (deferred — priority is per-source-globally).
- No deny-lists for "actively suppress this Wikidata field" (deferred — covered by per-field pinning instead).
- List endpoint cards do not surface source attribution.
## Current state (for context)
- `software_external_datas` already stores one row per `(externalId, sourceSlug)`, joined to a `sources` table that has a `priority` integer.
- `mergeExternalData.ts` priority-sorts external rows and deepmerges them.
- `createPgSoftwareRepository.ts:getDetails()` then layers manual fields from `softwares` on top via `?? extData?.x` fallbacks.
- The merged result is returned as a flat `Software` object. No provenance survives the merge.
- Manual fields live in dedicated columns on `softwares` (`url`, `image`, `description`, `keywords`, `programmingLanguages`, `license`, `latestVersion`, etc.).
- Today's array merge is inconsistent: `keywords` and `programmingLanguages` use override semantics; `applicationCategories` and `authors` use concat.
## Target architecture
### Schema changes
1. **Add a new `sources.kind` enum value: `user_input`.**
2. **Add a seeded row in `sources`** (only for opted-in deployments, see below) with:
- `slug = 'user_input'`
- `kind = 'user_input'`
- `priority = MAX(existing_priority) + 100` (highest, sparse for future inserts)
- No gateway implementation.
3. **Make `software_external_datas.externalId` nullable.**
4. **Add a partial unique index** `(softwareId, sourceSlug) WHERE externalId IS NULL` to enforce one user_input row per software.
5. **Drop dead content columns from `softwares`** *one release later, after the new code is stable*: `name`, `description`, `url`, `image`, `keywords`, `programmingLanguages`, `license`, `latestVersion`, `softwareHelp`, `codeRepositoryUrl`, `applicationCategories`, `operatingSystems`, `runtimePlatforms`. Keep catalog metadata: `id`, `addedByUserId`, `referencedSinceTime`, `isStillInObservation`, `dereferencing`, `customAttributes`, `parentSoftwareWikidataId`, joins.
### Read path
- One unified merge function. Priority-sort all `software_external_datas` rows for a software (including any `user_input` row), then for each field pick from the highest-priority row that has a value.
- **Scalars** (`name`, `description`, `url`, `image`, `license`, `latestVersion`, `codeRepositoryUrl`, ...): highest priority wins.
- **Arrays** (`keywords`, `programmingLanguages`, `applicationCategories`, `authors`, `identifiers`, `referencePublications`, `operatingSystems`, `runtimePlatforms`, `providers`): **union across all sources** with field-specific dedupe keys:
- `keywords`, `programmingLanguages`, `applicationCategories`, `operatingSystems`, `runtimePlatforms`: dedupe by lowercase string.
- `authors`: dedupe by `@id` if present, else by normalized name.
- `identifiers`: dedupe by `(propertyID, value)`.
- `referencePublications`: dedupe by DOI / `@id`.
- `providers`: dedupe by URL or name.
- Note: this is a **behavior change for `keywords` and `programmingLanguages`**, which previously used override semantics. They now union across all contributing sources.
- The old `?? extData?.x` chains in `getDetails` and the split between `mergeExternalData.ts` and `getDetails` post-processing both go away.
### API contract
- **List endpoint**: unchanged. Returns `Software[]` (the flat merged shape).
- **Detail endpoint**: returns `SoftwareDetail = Software & { dataBySource: SoftwareSourceData[] }`. The `dataBySource` array is ordered by source priority descending (winner first), and only contains entries for sources that actually have a row for this software.
- **`SoftwareSourceData` type**: a narrow content type mirroring one row of `software_external_datas`, plus discriminator fields:
- `sourceSlug: string`
- `priority: number`
- `lastDataFetchAt: string` (ISO date)
- For the `user_input` entry, this column doubles as "last edited at". A `lastEditedByUserId` may be added later if needed; for now, label-only difference in UI.
- Content fields: `name`, `description`, `url`, `image`, `codeRepositoryUrl`, `license`, `keywords`, `programmingLanguages`, `applicationCategories`, `authors`, `identifiers`, `referencePublications`, `providers`, `repoMetadata`, `operatingSystems`, `runtimePlatforms`, `latestVersion`.
- The new types are exported from `api/src/lib/index.ts` so the web package can import via `import type { SoftwareDetail, SoftwareSourceData } from "api"`.
### Write path
- Form pre-populates from the `user_input` row only — fields user_input doesn't have show as blank with a placeholder hint showing what other sources currently provide.
- Form save = UPSERT into `software_external_datas` for `(softwareId, sourceSlug='user_input')`. Empty inputs save as NULL → fall through to next-priority source on read.
- `INSERT ... ON CONFLICT (softwareId, sourceSlug) WHERE externalId IS NULL DO UPDATE` — Postgres handles create-or-update in one statement, no application branching.
- `createSoftware` flow: creates the catalog `softwares` row, optionally creates a `user_input` row in `software_external_datas` if any content fields were submitted, triggers external data fetch as today.
- `updateSoftware` flow: upserts the `user_input` row.
- The refresh job (`refreshExternalData.ts`) iterates registered gateways. `user_input` has no gateway, so it's naturally invisible to refresh — no special-casing needed.
### UI
- **Provenance drawer**: a slide-over panel triggered by an icon in the `HeaderDetailCard` of the software detail page. Layout: one card per source, in priority order (winner first). Each card shows source name, fetch/edit timestamp, and a key-value list of every field that source contributed. Sources with no row for this software do not appear.
- **Per-field "i" icon in the edit form**: opens a popover showing all sources' values for that single field, with a "use this value" button that copies into the input.
- **Both the drawer and the popover** are rendered by the same reusable component, e.g. ``. The drawer passes all fields, the popover passes one. Single source of truth for the rendering logic.
- **Inline placeholders** on form inputs show "Currently displayed: {value} (from {source})" so editors understand blank means "fall through".
- Drawer empty state: "No source data available yet" for brand-new softwares before any source has contributed.
### Per-deployment opt-in
- **The migration always runs the schema bits** (nullable `externalId`, partial unique index, new enum value).
- **The migration reads `api/src/customization/ui-config.json` directly** at run time, via the same mechanism `bootstrap.ts:20` uses (`import rawUiConfig from "../customization/ui-config.json"`). The Helm `api-deployment.yaml` mounts the ConfigMap at `/app/api/dist/src/customization/ui-config.json` before the container starts, so the file is in place when migrations run as part of `pnpm db:up` (called from `pnpm start`).
- **Conditional logic:**
```ts
const userInputEnabled =
uiConfig.home.usecases.editSoftware.enabled ||
uiConfig.home.usecases.addSoftwareOrService.enabled;
```
- **If true**: insert the `user_input` row in `sources`, then backfill — for each `softwares` row with at least one non-NULL content column, INSERT a `user_input` row in `software_external_datas` copying the non-NULL columns across (NULL stays NULL — don't fill with merged values). Rows with all-NULL content get no `user_input` row.
- **If false**: skip the seed and the backfill entirely. The deployment ends with the schema bits in place but zero `user_input` data. SILL gets the migration; pure-aggregator reusers are no-ops.
## Migration ordering
1. **Migration N**: schema bits + conditional source row + conditional backfill (one Kysely migration file).
2. **Code deploy**: read path reads from `software_external_datas` only. Write path upserts into `user_input` row. Legacy `softwares` content columns still present, untouched, available for emergency rollback.
3. **Migration N+1** (next release, after stability): drop the dead content columns from `softwares`.
Steps 1 and 2 are reversible without data loss. Step 3 is the only irreversible step.
## Acceptance criteria
- [ ] Detail endpoint returns `dataBySource: SoftwareSourceData[]` ordered by priority desc.
- [ ] Provenance drawer accessible from the software detail header, listing every contributing source with its full per-field contribution.
- [ ] Per-field "i" icon on the edit form opens a popover showing all sources' values for that field.
- [ ] Edit form pre-populates from `user_input` only; blank fields fall through to external sources after save.
- [ ] List endpoint behavior unchanged (same `Software` shape, same merged values).
- [ ] On a SILL-style deployment, post-migration the merged detail view is visually identical to pre-migration for every existing software.
- [ ] On a reuser deployment with `editSoftware.enabled: false` and `addSoftwareOrService.enabled: false`, no `user_input` row in `sources`, no `user_input` rows in `software_external_datas`, no UI changes.
- [ ] Array fields (including `keywords` and `programmingLanguages`) union across sources with field-specific dedupe.
- [ ] Refresh job (`refreshExternalData.ts`) does not touch `user_input` rows.
- [ ] One unified merge function; the `?? extData?.x` chain in `getDetails` is removed.
## Open / deferred
- **Late-flip case**: a deployment that runs the migration with the flag off, then flips it on later, won't auto-seed the `user_input` source row. Either add a lazy-seed-on-API-startup check, or a separate `pnpm db:seed-user-input` command. Decide when needed.
- **Deny-lists** for "actively suppress this Wikidata field": deferred. Editors who want to hide a wrong external value must currently override with a different value.
- **Per-field priority overrides**: deferred.
- **Edit history / audit log**: out of scope.
- **Exact placement** of the drawer trigger icon in `HeaderDetailCard`: design review during implementation.
- **`name` requirement at create time** when importing from a Wikidata Q-number with no manual content: confirm form validation allows deferring `name` to the external source.
## Files most likely to change
- `api/src/core/adapters/dbApi/kysely/migrations/.ts` — the migration described above.
- `api/src/core/adapters/dbApi/kysely/mergeExternalData.ts` — unified merge function.
- `api/src/core/adapters/dbApi/kysely/createPgSoftwareRepository.ts` — `getDetails` reads sources only, returns `dataBySource`.
- `api/src/core/usecases/readWriteSillData/types.ts` — new `SoftwareSourceData` and `SoftwareDetail` types.
- `api/src/core/usecases/updateSoftware.ts`, `createSoftware.ts` — write into `user_input` row.
- `api/src/lib/index.ts` — export new types.
- `web/src/ui/pages/softwareDetails/HeaderDetailCard.tsx` — add provenance drawer trigger.
- `web/src/ui/pages/softwareDetails/` — new `SourceProvenanceView` component (drawer + per-field popover).
- `web/src/ui/pages/softwareForm/SoftwareForm.tsx` — placeholder hints + per-field "i" icons.
## Implementation notes for an AI agent
- Keep migration steps 1 and 2 reversible — do not drop dead columns in the first migration.
- Use `INSERT ... ON CONFLICT DO UPDATE` for the user_input upsert; do not branch on row existence in application code.
- The merge function should be a single pure function taking `SoftwareSourceData[]` and producing the merged `Software` shape. Use it in both list and detail read paths.
- Field-specific dedupe keys live as small helpers (`mergeAuthors`, `mergeIdentifiers`, etc.) in the same file as the merge function — not as configuration.
- The migration's UI config read should mirror `bootstrap.ts:20` exactly. Don't invent a new env var.
- New types must be exported from `api/src/lib/index.ts` so the web package can import them.
Contributor guide
Assessment
This issue has not been assessed yet.