total coordinates

Nobody has claimed this yet.

problem statement

Although we store images as N-dimensional arrays, it's important to remember that images are made of individual measurements / samples, and that in many imaging modalities each sample has its own coordinate metadata. I'm going to use the term "total coordinates" to describe coordinate metadata for each sample of an image; this stands in contrast to coordinate metadata that assigns a single coordinate value to an entire image. OME-NGFF does not support total coordinates. Failing to do so means we are under-describing our images, which impairs the ability to properly analyze, share, and reproduce bioimaging datasets.

examples

I will illustrate this with a few examples:

Point-scanning microscopes acquire individual samples at different times, and at different axial positions (unless the field curvature is null). Properly mapping samples into physical coordinates requires encoding the acquisition time, and the axial position, of each individual sample.
sCMOS cameras often run in rolling-shutter mode, in which case rows of samples are acquired sequentially to form a single image. As with the above example, mapping samples into physical coordinates requires encoding the acquisition time of each individual sample.
Fast volumetric microscopy often requires a continuously moving objective lens, which means that each 2D plane is acquired across a varying axial interval (i.e., the first row and the last row of pixels are at different axial positions in any given image). Similar to the above examples, we need to encode the z position of each individual sample in order to relate our data to physical coordinates.

Note that these are not mutually exclusive scenarios -- a point-scanning system with field curvature my use a fast z-scanner for rapid 3D imaging. If speed demands it, we can make the point-scanning bidirectional, and the z-scanning as well, which means that temporally adjacent samples / planes are no longer spatially adjacent. Encoding "total coordinates" for this image would require storing 3 spatial coordinates and one temporal coordinate for each individual sample.

matrix transformations don't work

I note that matrix transformations completely fail here, because

matrix transformations would require all data to have bloated dimensions (a 2D image has to be promoted to 4D for a time + z axis embedding)
matrix transformations can't handle irregular temporal or spatial sampling, which are very real properties of image acquisition
Matrix transformations are basically functions, but for things like spatially varying field curvature or spatially varying illumination intensity, there is no guarantee that a functional form of the coordinate can be found. We will be stuck with data in these cases.

Unsatisfying alternatives include raster-scanning-specific metadata, field-curvature-specific metadata, etc, but this seems like a pretty ugly way to solve a general problem, and it's not easily extensible.