DiamondLightSource / DiamondLightSource/smartem-decisions
Image fingerprinting at intake for path-independent file matching
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
Atlas/grid-square image files are rsync'd from Win10 EPU workstations to the DLS filesystem by custom scripts that rewrite paths and filenames. As a result the absolute paths stored in the DB (`grid.atlas_dir`, `gridsquare.image_path`, `atlas.storage_folder`) frequently do not match where the files actually live at the destination. We currently see three path dialects in one prod dump (POSIX `/dls/...`, Windows `X:/...`, Windows backslash `Y:\...`), plus cross-proposal filing (atlas acquired under one proposal stored under another visit).
A path-based resolver (literal -> prefix remap -> fuzzy-by-id search) is the pragmatic bridge for already-ingested data, but it is fragile: EPU numeric ids are assigned per microscope/session with no guaranteed shared pool, so cross-scope id collisions are possible, and fuzzy search is heuristic.
## Goal
Fingerprint each image at intake (on the source workstation, before rsync rewrites anything) and persist the fingerprint in the DB, so files can later be matched at any location by content, fully decoupled from path. This is the durable, root-cause fix; the path cascade covers legacy data that has no fingerprint.
## Scope / subtasks
- [ ] **Choose a fingerprinting method.** Requirements: works on Win10 and Linux; filesystem-agnostic; image-format-agnostic (mrc, tiff, jpg, dm); performant on large files (a full content hash of every MRC/TIFF at intake is I/O-heavy). Evaluate cheap-but-robust options (e.g. size + partial hash of first/last N KB, xxhash/blake3, or a header+size composite). Bare EPU id alone is insufficient (collision risk above).
- [ ] **Agent change.** Compute the fingerprint at intake in the ingest path (`fs_parser` / `event_processor`), before rsync. Consider doing it incrementally as files are observed.
- [ ] **DB schema + migration.** Add fingerprint column(s) to the relevant tables (gridsquare, grid/atlas, possibly foilhole). Alembic migration.
- [ ] **Backend resolver integration.** Add fingerprint match as the most reliable tier of the image-path resolver cascade. Side effects: if the fingerprint is exposed in the API, that is an API-schema change (version bump + FE client regen); if kept internal, no API change.
- [ ] **ADR** in smartem-devtools decision-records documenting the chosen method and the resolver-cascade-vs-fingerprint relationship.
## Context
Reverse-engineered from a prod dump + matching imagery on 2026-06-08. The path-resolver cascade (literal -> prefix -> fuzzy-by-numeric-id, scoped to the acquisition's own DLS subtree to avoid cross-scope collisions) is the legacy bridge; this issue is the durable replacement for new data. The two coexist.
---
## Status note (2026-09-02): the legacy bridge does not exist either
Re-checked against `main`. Neither half of the plan described above is implemented:
- **No fingerprinting.** There is no `fingerprint`, `checksum`, `content_hash`, `xxhash` or
`blake3` anywhere in `src/`.
- **No path-resolver cascade.** This is the important one, because the text above refers to the
cascade as though it were current state. It is not. Image serving uses the stored path
directly:
```python
return await _cached_image_response(Path(gridsquare.image_path), None)
```
There is no literal-then-prefix-then-fuzzy tiering, no prefix remapping, and no scoping to the
acquisition's subtree.
**Consequence.** Image serving currently depends on the paths in the database being literally
correct on whichever host serves them. That holds in local development only because the `/dls`
tree is mounted so the stored paths happen to resolve. It cannot hold for production data, where
this issue itself documents three path dialects in a single dump (POSIX `/dls/...`, Windows
`X:/...`, Windows backslash `Y:\...`) plus atlases filed under a different proposal from the one
they were acquired under.
**Scope correction.** This issue is now the only record that *either* mechanism is needed, so it
covers both, in order:
1. **Path-resolver cascade** - the bridge for already-ingested data, which has no fingerprint and
never will. Literal match, then prefix remapping across the known dialects, then
fuzzy-by-numeric-id scoped to the acquisition's own subtree to avoid cross-scope collisions.
Fragile by design, and acknowledged as such above, but it is what makes existing data
serviceable.
2. **Fingerprinting at intake** - the durable fix for new data, scoped as originally written
above.
They coexist as originally stated; the point of this note is that neither is built, and the
first is the prerequisite for anything that reads existing imagery.
**Priority context.** Improving front-end image visualisation presupposes that images resolve at
all. Optimising the delivery of images the backend cannot locate is the wrong order of work, so
this should be settled before, or alongside, that effort rather than after it.
Contributor guide
Assessment
This issue has not been assessed yet.