darktable-org / darktable-org/darktable
Improve Lua integration for external masks
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 13.1k
- Forks
- 1.4k
- Avg merge
- 22h 14m
- Merged PRs (30d)
- 198
Description
Is your feature request related to a problem? Please describe.
I created a Lua script to use Segment Anything Model 2 (SAM2) from Meta AI from Darktable. The script appears in the darkroom as “SAM2” and triggers an external Python command such as:
sam2-box -i "/tmp/DSC04927.png" -o "/tmp" -m 1 -n 1 --pfm
Where "-i" is the input file "-o" the output dirrectory, "-m" the modle, "-n" the number of mask to generate and "--pfm" the file format (pfm or png). This command opens a separate OpenCV window where I draw a bounding box. SAM2 then generates mask files that I later import into Darktable using the external raster mask module.
This workflow works, but it has several problems:
- It relies on an external window to draw the selection box, and don't feel integrated into Darktable.
- The Lua API does not provide a way to draw a bounding box directly in Darktable and obtain its coordinates.
- The Lua API does not allow importing a newly generated raster mask or selecting the folder for the external raster mask module.
- The external raster mask module imports all masks from a folder, making it difficult when you generate masks for more than one picture.
- The module only accepts pfm files. They can be large, and png support would be useful for users who prefer a smaller format.
Describe the solution you'd like
I would like several enhancements to improve integration for external mask-generation workflows:
- A Lua API function to allow users to draw a rectangle or bounding box directly in Darktable’s darkroom and return its coordinates. This would let us run command such as:
sam2-box -i "/tmp/DSC04927.png" -o "/tmp" -m 1 -n 1 --pfm -s 0 430 1600 700
Where "-s 0 430 1600 700" is the box boundary (x1=0, y1=430, x2=1600 and y2=700)
- Lua API access to import a raster mask file or at least select the source folder for the external raster mask module.
- A way for the external raster mask module to filter which files to import (for example, by filename pattern) so a folder can contain masks for multiple images without conflict.
- Support for png mask files in the external raster mask module in addition to pfm.
Alternatives
The current workflow works, but it is not user‑friendly. I have to rely on an external window to draw the bounding box, which breaks the Darktable workflow and UI integration. I also have to manually filter and import the generated masks with the external raster mask module. Another issue is that pfm files are about 100 times larger than equivalent png, making storage and performance much less efficient.
Additional context
Here is the Python script I am running inside a virtual environment to generate the masks:
https://github.com/AyedaOk/sam2-tools
And here is the Lua script that integrates this into Darktable (SAM2.lua):
https://github.com/AyedaOk/DT_custom_script
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the referenced SAM2.lua script and darktable’s Lua API and external raster mask module entry points. The request covers several separate enhancements, so first identify and scope one integration change, then define completion around that change with corresponding coverage for mask import or selection behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- lua, opencv, python
- Domain
- computer-vision, desktop
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100