Feature Request: Expose dt_metadata_add_metadata to Lua for script-defined per-image metadata

Is your feature request related to a problem? Please describe.

Lua scripts have no good way to persist structured per-image state across darktable restarts. For example, a publish/export plugin (such as one targeting Flickr, SmugMug, or any remote service) needs to remember, per local image, the remote photo ID it was uploaded as and which remote album(s) it belongs to. Today the only Lua-reachable persistent per-image store is image tags — a flat string namespace that's awkward and collision-prone for structured data. darktable's C core already has custom metadata fields (dt_metadata_add_metadata()), which are exactly the right shape, but there is no Lua binding to register one.

Describe the solution you'd like

Add a Lua binding darktable.register_metadata{ ... } that lets a script declare a persistent, per-image, named metadata field — a thin Lua wrapper around the existing C dt_metadata_add_metadata().

darktable.register_metadata{
  tagname = "Xmp.dtpublish.photo_id",  -- full XMP tag; last segment becomes the property name
  name    = "Remote photo ID",         -- label in the metadata editor
  visible = false,                     -- optional: hide from the editor
  private = true,                      -- optional: keep out of exported XMP
}

-- thereafter, on any dt_lua_image_t:
image.photo_id = "53812345678"   -- write
local id = image.photo_id        -- read

Nearly all the machinery already exists: definitions persist in data.meta_data, and the generic metadata_member getter/setter resolves fields dynamically — so this is essentially a thin, idempotent binding plus exposing the per-field member registration. It gives scripts a structured, schema'd per-image store instead of encoding state into tag strings.

Useful well beyond any one plugin: publish/export targets (remote photo IDs, album membership), reverse-geocode caches, AI-tagging confidence/model, external-DAM sync UUIDs + timestamps, and rating bridges.

Note — this needs a new field of the script's own, which is why the metadata editor can't substitute. The field a script wants is its own (e.g. Xmp.dtpublish.photo_id), so it doesn't collide with the user's real metadata — not an existing tag. The metadata-editor "add tag" dialog only lists tags from exiv2's known-tag database (dt_exif_get_exiv2_taglist()), so a user can add existing tags but cannot create a new one of the script's own — and it pushes a manual setup step onto every user. dt_metadata_add_metadata accepts any tag name, so a small Lua binding to it is the only way for a script to create its own field.

Describe alternatives you've considered

• Image tags — the only Lua-reachable persistent per-image store today. Works, but a flat string namespace: encoding structured/multi-value state (numeric ID + album set + timestamp) is awkward, collision-prone, has no schema, and pollutes the user's tag cloud.
• The built-in notes field (dt_lua_image_t.notes) — persistent and writable from Lua, but it's a single user-facing field (clobbers the user's own notes) and single-valued, so it can't hold structured per-service state.
• darktable.preferences — global / per-script only, with no per-image variant; can't map a value to an individual image.
• Have the user add the field in the metadata editor — the editor's add-tag dialog only offers exiv2-known tags (dt_exif_get_exiv2_taglist()), so a script can't get a new field of its own (Xmp.<plugin>.*) this way; it only covers reusing existing tags, and forces a manual setup step on every user.
• A C companion / patched build — dt_metadata_add_metadata() is already callable from C, but requiring a custom-compiled or forked darktable defeats the point of a drop-in Lua script. This request is exactly to make that capability reachable from Lua so no fork is needed.
• External sidecar DB maintained by the script — possible, but reinvents per-image storage darktable already has, and doesn't travel with the image or integrate with XMP.

Additional context

Full design proposal below, with source citations. All file:line references are to the current master.

---

Existing machinery (what already works)

These are the load-bearing facts a reviewer can confirm cheaply.

1. Definitions persist in data.meta_data (shared, survives restart). dt_metadata_add_metadata() (src/common/metadata.c:60) INSERTs the field definition into data.meta_data:

INSERT INTO data.meta_data
  (key, tagname, name, internal, visible, private, display_order)
  VALUES(NULL, ?1, ?2, ?3, ?4, ?5, ?6)

key is passed NULL, so SQLite auto-assigns it; the function then reads it back (metadata.c:80-84) and prepends the entry to the in-memory _metadata_list. data.meta_data lives in data.db (shared), distinct from the per-image values table main.meta_data. On next launch, dt_metadata_init() (metadata.c:189) reloads every definition. No new storage code is needed for persistence.

2. The generic getter/setter resolves the field at call time. metadata_member() (src/lua/image.c:312) backs every Lua image.<field> metadata accessor and resolves the field dynamically — read via dt_metadata_get_lock (image.c:319), write via dt_metadata_set (image.c:331). Because the key is looked up at call time, a newly-added field needs no new read/write code.

3. The dt_metadata_t struct already carries every option to expose (src/common/metadata.h:25-34): tagname, name, internal, visible, priv, display_order (+ DB-assigned key).

Proposed Lua API

Modeled on darktable.preferences.register (a single function taking a named-style argument list).

Lua option	dt_metadata_t field	Default	Notes
tagname	tagname	—	Required. Full XMP tag, e.g. Xmp.dtpublish.photo_id; must contain a . (the Lua property name is the last segment — see G2).
name	name	—	Required. Human label in the metadata editor.
visible	visible	true	Editor visibility (also participates in the export predicate — see semantics).
private	priv	false	"Do not export" — suppresses the field from exported XMP.
display_order	display_order	0	Sort order in the editor list.
(forced)	internal	false	Hard-coded false; a field with internal = true would be invisible to Lua. Not exposed.
(not exposed)	key	—	Assigned by SQLite.

"Hide from the editor" is visible = false; "keep out of exports" is private = true.

Semantics: internal vs visible vs priv

Flag	What it gates	Evidence
internal	Master gate. When true, the field is excluded from the Lua image type (src/lua/image.c:627 registers metadata_member only if(!metadata->internal)), from the editor grid (src/libs/metadata.c:812-813), from the editor prefs list (metadata.c:868), and from XMP export (src/common/exif.cc:5835). A Lua-registered field must have internal = false.
visible	Editor visibility (src/libs/metadata.c:479, :527). Does not affect Lua access. Does affect export: a field is written to XMP iff visible && !priv (src/common/exif.cc:5051, :5835).
priv	"Private" / do-not-export (private column, commented // do not export at src/libs/metadata.c:858). Export requires !priv. Does not affect Lua access.

⚠️ Key subtlety: because export tests visible && !priv, setting visible = false also keeps the field out of exported XMP. So for an opaque plugin bookkeeping field (e.g. a remote service's photo ID), visible = false alone gives "invisible in the editor and never exported".

Design gotchas the implementation must handle

G1 — Idempotency. dt_metadata_add_metadata() INSERTs unconditionally with no existence check (metadata.c:60-94). Since Lua scripts run on every launch, the binding must guard with dt_metadata_get_metadata_by_tagname() (metadata.c:107) and no-op if the field already exists, or it will accumulate duplicate rows.

G2 — Subkey collision. The Lua property is only the substring after the last . (dt_metadata_get_tag_subkey, metadata.c:174), and lookups match only that segment, returning the first hit (dt_metadata_get_key_by_subkey, metadata.c:151-172). So Xmp.darktable.notes and Xmp.acdsee.notes would both map to image.notes and collide. The binding should reject a tagname with no ., and reject one whose final segment already resolves to a different existing tagname.

G3 — The internal/visible overload. internal is the only flag controlling Lua visibility. Don't let callers set internal = true. "Hide from editor" = visible; "do not export" = priv.

G4 — Member-registration timing & the static qualifier. metadata_member is static int metadata_member(lua_State *L) (image.c:312), so a new translation unit can't reference it. The image-type member loop runs once at Lua init (image.c:621-633); a field added at script-run time is only usable if a member for its subkey is registered on dt_lua_image_t via dt_lua_type_register. Preferred fix: un-static metadata_member (or add an exported dt_lua_image_register_metadata_member(L, subkey) helper) and call it after a successful add, so the field works in the same session.

Who else benefits

• Publish/export plugins (e.g. Flickr, SmugMug, Piwigo-via-Lua) — remote photo IDs + album membership.
• Reverse-geocode caches — resolved place name / OSM IDs per lat-long, looked up once.
• AI / auto-tagging — model name, version, per-image confidence.
• External-DAM sync — foreign UUID + last-synced timestamp.
• Rating / flag bridges — mirror an external app's rating without clobbering darktable's native one.

Minimal patch checklist

1. New binding darktable.register_metadata (new src/lua/metadata.c or extend src/lua/image.c): validate tagname/name; G1 idempotency guard; G2 collision guard; build a dt_metadata_t with internal = FALSE; call dt_metadata_add_metadata; register the image-type member (needs G4).
2. G4 plumbing — un-static metadata_member or add an exported registration helper in src/lua/image.c.
3. API-version bump so scripts can feature-detect.
4. luadocs page for darktable.register_metadata; cross-link from dt_lua_image_t.
5. Register the new module in the Lua bootstrap if a new file is added.

Backward compatibility

Scripts should feature-detect and fall back to tags, never hard-depend on this:

if type(darktable.register_metadata) == "function" then
  darktable.register_metadata{ tagname = "Xmp.dtpublish.photo_id", name = "Remote photo ID",
                               visible = false, private = true }
  -- use image.photo_id
else
  -- fallback: encode state in a namespaced tag, e.g. "darktable|publish|id|<photo_id>"
end

Validation log — every claim above checked against source

#	Claim	Status	Evidence
1	dt_metadata_add_metadata persists the definition by INSERT into data.meta_data (distinct from per-image main.meta_data); key auto-assigned.	VERIFIED	INSERT data.meta_data VALUES(NULL, ...) src/common/metadata.c:66-69; key read back metadata.c:80-87; reload dt_metadata_init metadata.c:189-225.
2	Generic metadata_member resolves the key dynamically; new field needs no new read/write code.	VERIFIED	dt_metadata_get_key_by_subkey src/lua/image.c:315; read image.c:319; write image.c:331.
3	Startup loop registers image.<subkey> only when !internal; hiding from UI uses visible=false.	VERIFIED (clarified)	!metadata->internal src/lua/image.c:627; visible editor src/libs/metadata.c:479,527; export visible && !priv src/common/exif.cc:5051,5835. Clarification: visible=false also suppresses export.
4	Lua property name is only the last dotted segment; shared final segments collide.	VERIFIED	dt_metadata_get_tag_subkey metadata.c:174-178; member named image.c:630; first-match lookup metadata.c:151-172.
5	dt_metadata_add_metadata has no existence check → duplicate rows unless guarded.	VERIFIED	No dedup metadata.c:60-94; guard dt_metadata_get_metadata_by_tagname metadata.c:107.
6	metadata_member is static; patch must expose it or relocate registration.	VERIFIED	static int metadata_member src/lua/image.c:312; registration image.c:629-630.
7	No existing Lua binding registers metadata.	VERIFIED	Zero dt_metadata_add_metadata matches under src/lua; only callers are src/libs/metadata.c:1068 and the decl/def.

Net result: all claims verified. The only refinement (Claim 3): internal is the sole gate on Lua visibility, but visible is overloaded — it gates editor display and participates in the export predicate (visible && !priv), so visible = false also keeps a field out of exported XMP.

Start with src/common/metadata.c and metadata.h, then trace metadata_member and the image-type registration loop in src/lua/image.c. Review the cited Lua registration patterns and luadocs before choosing the binding location. Done means idempotent registration, collision validation, same-session image access, API-version coverage, and documentation.