RFC: where to store soft alpha masks – XMP or a second sidecar?

I want to work on a few mask features and they all hit the same wall, so I would rather agree the approach before writing code.

• One-click subject mask – select the subject without drawing anything.
• Depth masks – a model gives a per-pixel depth map and you pick a range over it.
• A raster brush – not AI, and just an idea now. The vector brush is poor for painting, and a raster one becomes easy once we can store pixels.
• Inpainting patches – further off, same shape of problem.

All of them produce per-pixel data that cannot be turned into vectors without losing exactly the quality we want – hair, fur, glass. The existing object tool (SAM 2.1) avoids this by tracing its result into DT_MASKS_PATH forms, which works because SAM gives a clean contour. Subject masks and depth maps do not.

So the question is whether we are willing to store binary pixel data at all, and if so, where. If the answer is no, the rest does not matter and I would rather know now.

Only the pixels are at issue. The mask's description is a fixed-size struct like any other form, so it slots into masks_history.points and reaches the XMP through the gz+base64 encoder we already have (exif.cc:3382) at no cost.

How big, really

Both models emit fixed-size output, so this is not open-ended. Stored as uint8 and deflated:

	native output	deflated	base64 for XMP
depth (depth-da2-small)	518x518	17 KB	22 KB
subject (mask-subject-birefnet)	1024x1024	219 KB	292 KB

Two thresholds matter. 100 KB is where SQLite's own measurements say a blob stops being faster inside the database than in a separate file. 64 KB is the JPEG segment limit – the .xmp sidecar itself has no size limit, but dt_exif_xmp_attach_export() silently drops history from exported JPEGs when the XMP will not fit (exif.cc:6300).

Depth is under both at native size. A subject mask at native size is over both.

That is a fact about these two models, not about depth and masks. depth-da2-small is the smallest DepthAnythingV2 variant and the family scales up at test time, so a future depth model could easily land where BiRefNet is now. Which is the argument for deciding by size rather than by feature: a rule that says "depth in the XMP, subject masks elsewhere" is wrong the day we ship a bigger depth model, while a rule that says "over 100 KB goes elsewhere" keeps working.

Two caveats on the numbers. The 219 KB is a synthetic mask at BiRefNet's size rather than a real inference, so treat it as a floor. And uint8 is doing real work: the same depth map at uint16 is 310 KB, not 17 KB.

What can be regenerated, and what cannot

	regenerable?
depth, subject, SAM embeddings	yes, from model + input + pipeline state
brush strokes	no
inpainting output	no – the models are stochastic

This decides more than the file format does. Painted work cannot be recovered at all if it is lost, which rules out the mipmap-cache pattern for it: that cache lives in a directory named after a hash of library.db's absolute path, with files named by imgid (mipmap_cache.c:362, :609), so moving a library or rebuilding it from XMPs orphans every entry. Harmless for thumbnails. Not acceptable for a brush stroke.

But "regenerable" is weaker than it sounds, and I would not treat the first group as a pure cache either. Regenerating needs the same model, still installed, still the same version – and a model update produces a different mask, not the same one again. So a mask that is thrown away and re-inferred later is not guaranteed to be the mask the user approved. Storing it keeps the edit as it was made, which is what the XMP does for everything else.

The practical difference is transferability. Copy a folder of photos to another machine and a cached mask is gone, so every image re-runs inference – if the model is even installed there. A mask stored beside the photo simply arrives with it. That is the argument for .dtdata over a cache directory even for data we could technically regenerate.

What Lightroom did

Worth knowing, because Adobe hit this exactly. They shipped AI masks in the catalog and the XMP, it bloated, and since Lightroom Classic 15.0 they write a second binary sidecar, .acr – but only when an image has heavy edits, so most photos still get one sidecar.

Two lessons. Their .acr is undocumented and unreadable, so nobody outside Adobe can debug it. And their rule for when to write one is vague enough that there is an open bug about sidecars several times the size of the image they sit beside, for images whose edits are already baked into their pixels and cannot use the data.

Options

Whether to store pixel data at all. If not, most of these are not built. Depth has no contour to trace, a raster brush is raster by definition, and inpainting output is pixels rather than a shape – none of the three has a vector form to fall back to. Only the subject mask does, and that fallback already exists: it is the object tool.

Where an oversized entry lives.

	keeps	costs
blob in library.db	no new files, works with sidecars off	VACUUM rebuilds it, every schema upgrade copies it, past SQLite's own 100 KB crossover
file in a cache directory	fast, nothing beside the photos	orphans when a library moves or is rebuilt from XMPs, and re-inference needs the same model version installed
second sidecar next to the image	travels with the photo, survives a rebuilt database	a new file type for users to know about, and a zip dependency

What decides where an entry goes. By size, with a threshold – 100 KB is SQLite's own crossover and sits just above the 64 KB JPEG limit. Or by feature, "depth in XMP, subject masks outside", which is simpler to explain but wrong the day a bigger depth model ships. Or by image, as Adobe does, which is what produced the oversized sidecars above.

What happens when sidecars are off or the folder is read-only. Either the feature is unavailable, or regenerable data falls back to a cache and authoritative data to a database blob, or everything falls back to the database and we accept the growth for those users.

	sidecars on	sidecars off (never)
regenerable	second sidecar	cache; re-infer if orphaned
authoritative	DB blob + sidecar mirror	DB blob

The database always answers, so a preference never disables a feature and never silently discards painted work. Concretely the blob would be a table in library.db alongside masks_history, keyed by image and entry, with the same cascade so deleting an image cleans up its rasters. SQLite's 100 KB figure is a read-performance crossover rather than a limit – op_params blobs are already arbitrary size – so an oversized blob works, it is just not optimal. What scales badly is VACUUM main rebuilding the file and dt_database_backup() copying it on every schema upgrade, and this case is the intersection of two uncommon things: sidecars off and heavy use of painted masks, which are rare by nature because each one costs manual work. If it accumulates anyway, a third attached database isolates it – darktable already attaches data.db beside library.db with its own file, lock and version.

Whether the format separates cache from authoritative. If it does not, a preference that disables binary storage will one day silently discard painted work. If it does, that has to be in the format before a raster brush ships rather than retrofitted.

Container format. A zip with a small manifest, or something custom. Adobe's .acr is undocumented and unreadable, so nobody outside Adobe can debug it.

---

For what it is worth I would pick: yes to storing pixels, second sidecar for anything over 100 KB, per entry rather than per image, database as the fallback that always answers, and both entry classes in the format from the start.

Storage resolution is a separate question that changes the numbers rather than the design: BiRefNet's 1024 output stored at 512 is 28 KB instead of 219 KB and would keep most masks in the XMP entirely, at whatever quality guided upsampling gives back.

Start with masks_history and the XMP path in exif.cc, including exif.cc:3382 and dt_exif_xmp_attach_export() at :6300. Review the existing cache behavior in mipmap_cache.c and the proposed library.db storage options. Done means agreeing on whether pixel data is stored, its authoritative and cache forms, placement threshold, fallback behavior, and container format.