Add metadata: image size and checksum to help identify original image, also useful for large collections (#2)

This is a re-opened issue as the original https://github.com/darktable-org/darktable/issues/2492 was closed some time ago.

• @wpferguson indicated I should create a new issue
• Darktable contributor nodes indicate discussing changes prior to making a PR. My intention is to make a PR following the details described below. This is easier now with AI help but clearly it makes sense to propose and implement something that is acceptable to the owners of this repo.

The original premise of this proposal still stands

Why?

• Renaming files or reorganizing directories over time makes it hard to be
  sure a .xmp sidecar still matches its image — the sidecar only records
  a filename, not the image's identity.
• Storing size + checksum in both the DB and the sidecar fixes that: you
  can find the original image a .xmp relates to even after a rename,
  matching on content instead of filename. Not possible today.
• Matters most exactly where manual tracking breaks down: large
  collections, tens or hundreds of thousands of images. Secondary benefit:
  the same data enables duplicate-image detection.

What gets computed

• Size: stat(), free.
• Checksum: SHA-1, chosen for (i) properly identifying/verifying the
  underlying image and (ii) potential deduplication detection between
  different images — not a security/integrity context, so SHA-1's known
  collision weaknesses don't apply here. Reuses images.sha1sum — column
  exists, defined but not used by current code
  (src/common/database.c#L175);
  only other hits are historical CREATE TABLE migration steps and one
  unrelated comment in
  mipmap_cache.h#L112.
• Cost: measured 60MB file (typical raw size) hashes in ~70ms on a modest
  mobile CPU with SHA-NI (Ryzen 5560U, openssl speed sha1 ~1.1GB/s).
  Reading the same 60MB off NVMe: ~9-17ms (published figures, Gen3/Gen4).
  So hashing alone costs more than the raw disk read, but is a modest
  fraction (~10-30%) of the full first-read-and-decode operation, since
  demosaic/pixelpipe decode costs hundreds of ms. Answers parafin's perf
  objection in the original issue thread.

When computed

• Not at import time.
• Lazily on first full read, via
  dt_imageio_open()
  — single chokepoint for every format loader (rawspeed, libraw, tiff,
  magick fallback), already does a file-exists check.
• Lazy-only misses images never opened in darkroom, so it's not the
  primary path to a fully-checksummed library — see Backfill below.
• Two-stage write: populate the DB column immediately on compute; only
  write into the .xmp sidecar at the next natural xmp-write point —
  any
  dt_image_cache_write_release(img, DT_IMAGE_CACHE_SAFE),
  which already calls dt_image_synch_xmp(). No new write trigger needed; this path is
  already used at dozens of call sites (history edits, tagging, styles,
  darkroom exit, Lua edits).

Backfill of full collection

Deliberately out of scope for this proposal. Lazy on-load computation doesn't reach
images never opened in darkroom, so a separate mechanism is needed to
checksum an existing library in bulk.

I plan to write such a command-line driven utility to do that. Either way new sidecars files and the database will get updated as changes are made. Older image entries won't initially have that data.

Preference / opt-in

• New preference, e.g. plugins/darkroom/compute_checksum (bool), default:
  off.
• Off: nothing computed, no xmp changes, fully backward compatible with current version.

Check when loading if size/checksum changes and warn user, optional

Separate, optional feature — could be dropped from an initial version
without affecting the core proposal above.

• If a stored value exists and the file no longer matches on reload: warn,
  don't block, and update the stored value to the newly-detected one.
  Detects a file replaced/corrupted since it was last checksummed —
  useful feedback to the user, catching a case that currently can't be
  detected at all.
• Size mismatch: cheap stat(), checked on every load. Checksum mismatch:
  checked on a full read that's already happening, or immediately if the
  cheap size check already found a mismatch.
• Gated by the same opt-in preference.

XMP storage

• New optional darktable: namespace attributes:
  darktable:image_checksum="sha1:<hex>",
  darktable:image_size="<bytes>".
• Safe: darktable reads xmp by key lookup (xmpData.findKey(...)), not
  full validation, and the whole read path is wrapped in
  try { ... } catch(const Exiv2::AnyError &e)
  (dt_exif_xmp_read).
  Unknown attributes are ignored; old and new darktable versions
  interoperate on either sidecar format.

DB schema

• sha1sum CHAR(40) exists, unused.
• size/length column doesn't exist — needs a schema migration + version
  bump.
• Migration mechanism: a db_info table stores an integer version per db
  file; _upgrade_library_schema_step() runs steps on open if older,
  darktable refuses to open a db newer than it understands.
  CURRENT_DATABASE_VERSION_LIBRARY = 57 currently.

Open items

• Hash the raw file bytes directly (second read pass) vs. hook into
  whichever loader's own read buffer, to avoid redundant I/O.
• Exact xmp attribute formatting/ordering to match darktable's writer.
• Suggested improvement (to consider): store the binary digest in
  sha1sum rather than its hex string, halving the column's size. SQLite
  supports this with no schema change. Add a column comment noting the
  value is binary, not hex.

Feedback

Looking for feedback if this approach looks reasonable and if there are other things I should be considering.
Impact is intended to be minimal. The separate backfill process does not prevent new users, with this setting enabled, being able to get the sidecar files with valid size and checksum values allowing for improved collection management, verification, especially if someone wants to heavily reorganise their photo collection.

Start at src/imageio/imageio.c around dt_imageio_open(), then trace database handling in src/common/database.c, XMP synchronization through src/common/image_cache.c, and XMP parsing in src/common/exif.cc. Review the existing schema migration path and sha1sum column before resolving the open questions about hashing and XMP formatting. Done means an agreed, opt-in implementation that stores image size and checksum in the database and sidecars without changing behavior when disabled.