Background updating of sidecar files to speed up start time

Is your feature request related to a problem? Please describe.

On Linux I am using darktable v5.4.1 and when starting it up I see the "updating sidecar files" message appearing which may stay there for 2-3 minutes while scanning my photo archive.

Details of my archive for reference:

user@host:/data/photos$ find . -type f | grep -iE '\.(CR[23]|jpg|jpeg)' | wc -l
486557
user@host:/data/photos$ find . -type f | grep -iE '\.(xmp)' | wc -l
98309
user@host:/data/photos$ df -h /data1/photos
Filesystem                      Size  Used Avail Use% Mounted on
server:/photos  3.8T  2.9T  898G  77% /data1/photos

So nearly 500k photos, 100k known by darktable (not all directories are imported) and total disk usage about 3 TB.

I find the wait rather frustrating.

Describe the solution you'd like

Whatever I do in darktable most of these xmp files will not be touched or modified. Most of the time this is just a "verification", likely to have small changes. Therefore I think it would be convenient for darktable to run this processing in the background, and as files are noticed to have changed to do required processing (which I guess is to update the sqlite backend) appropriately. This gives a better user experience and would match some background processing done in other parts of the program. If photos that are currently being viewed are affected by .xmp file updates then update the "user's view" or "save the user's view" back to the .xmp file if it's been updated. Exact behaviour may need to be a policy to avoid unexpected overwrites when the .xmp file changes and the user has also made local changes. (???)

Alternatives

Alternatively be able to trigger darktable to do a metadata rescan / refresh from the GUI when needed.

Additional context

My photos are on a NAS accessed over ethernet and performance is generally good. It looks as if the scanning may be happening sequentially. It is quite possible that the scan speed could be improved by doing more work (scanning more files) in parallel.

In any case I would like to say thanks for darktable. It's an excellent tool and covers almost all of my photo needs. Thank you.

Start by locating the startup path that displays the "updating sidecar files" message and the sidecar-to-SQLite update processing. Review how the GUI currently handles metadata rescans and files being viewed. Done should include an agreed policy for background updates, changed-file handling, and conflicts with local edits, with behavior validated on large archives and NAS storage.