Some duplicate_action settings are not thread safe
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 2.1k
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 31
Description
When importing a ~1000 track collection in to beets I experienced the following error.
My guess is this is due to using `duplicate_action: merge` whilst having `threaded: yes` as when I set `threaded: no`, the problem went away (albeit it took much longer to import).
For example, consider a compilation that is imported from the following directory structure:
```
/root/
- foo/foo.mp3
- bar/bar.mp3
- zar/zar.mps
```
And then during the import, beets attempts to merge it with multiple threads in to
```
/root/
- compilation/bar.mp3
- compilation/foo.mp3
- compilation/zar.mps
```
Then my guess is that perhaps during the merge, threads can conflict on the ID of a track.
If this is too complicated to fix then I think the documentation should atleast be updated stating certain actions are not thread safe: https://beets.readthedocs.io/en/stable/reference/config.html#duplicate-action
### Problem
```
The content of stderr is 'Traceback (most recent call last):
File "/lsiopy/lib/python3.11/site-packages/beets/util/functemplate.py", line 574, in substitute
res = self.compiled(values, functions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/util/functemplate.py", line 598, in wrapper_func
args[VARIABLE_PREFIX + varname] = values[varname]
~~~~~~^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/library.py", line 428, in __getitem__
value = self._get(key)
^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/library.py", line 415, in _get
if self.for_path and key in self.album_keys:
^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/util/__init__.py", line 1081, in wrapper
value = func(self)
^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/library.py", line 396, in album_keys
if self.album:
^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/library.py", line 409, in album
return self.item._cached_album
^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/library.py", line 581, in _cached_album
self.__album.load()
File "/lsiopy/lib/python3.11/site-packages/beets/dbcore/db.py", line 562, in load
assert stored_obj is not None, f"object {self.id} not in DB"
^^^^^^^^^^^^^^^^^^^^^^
AssertionError: object 8 not in DB
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/lsiopy/bin/beet", line 8, in
sys.exit(main())
^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/ui/__init__.py", line 1285, in main
_raw_main(args)
File "/lsiopy/lib/python3.11/site-packages/beets/ui/__init__.py", line 1272, in _raw_main
subcommand.func(lib, suboptions, subargs)
File "/lsiopy/lib/python3.11/site-packages/beets/ui/commands.py", line 973, in import_func
import_files(lib, paths, query)
File "/lsiopy/lib/python3.11/site-packages/beets/ui/commands.py", line 943, in import_files
session.run()
File "/lsiopy/lib/python3.11/site-packages/beets/importer.py", line 340, in run
pl.run_parallel(QUEUE_SIZE)
File "/lsiopy/lib/python3.11/site-packages/beets/util/pipeline.py", line 446, in run_parallel
raise exc_info[1].with_traceback(exc_info[2])
File "/lsiopy/lib/python3.11/site-packages/beets/util/pipeline.py", line 358, in run
self.coro.send(msg)
File "/lsiopy/lib/python3.11/site-packages/beets/util/pipeline.py", line 170, in coro
task = func(*(args + (task,)))
^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/importer.py", line 1566, in manipulate_files
task.manipulate_files(
File "/lsiopy/lib/python3.11/site-packages/beets/importer.py", line 757, in manipulate_files
item.move(operation)
File "/lsiopy/lib/python3.11/site-packages/beets/library.py", line 915, in move
dest = self.destination(basedir=basedir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/library.py", line 979, in destination
subpath = self.evaluate_template(subpath_tmpl, True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/dbcore/db.py", line 625, in evaluate_template
return template.substitute(self.formatted(for_path=for_path),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/util/functemplate.py", line 576, in substitute
res = self.interpret(values, functions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/util/functemplate.py", line 568, in interpret
return self.expr.evaluate(Environment(values, functions))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/util/functemplate.py", line 256, in evaluate
out.append(part.evaluate(env))
^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/util/functemplate.py", line 163, in evaluate
if self.ident in env.values:
^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 780, in __contains__
File "/lsiopy/lib/python3.11/site-packages/beets/library.py", line 428, in __getitem__
value = self._get(key)
^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/library.py", line 415, in _get
if self.for_path and key in self.album_keys:
^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/util/__init__.py", line 1081, in wrapper
value = func(self)
^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/library.py", line 396, in album_keys
if self.album:
^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/library.py", line 409, in album
return self.item._cached_album
^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.11/site-packages/beets/library.py", line 581, in _cached_album
self.__album.load()
File "/lsiopy/lib/python3.11/site-packages/beets/dbcore/db.py", line 562, in load
assert stored_obj is not None, f"object {self.id} not in DB"
^^^^^^^^^^^^^^^^^^^^^^
AssertionError: object 8 not in DB
```
My configuration (output of `beet config`) is:
```yaml
plugins: web
directory: /music
library: /config/musiclibrary.blb
art_filename: albumart
threaded: yes
original_date: no
per_disc_numbering: no
convert:
auto: no
ffmpeg: /usr/bin/ffmpeg
opts: -ab 320k -ac 2 -ar 48000
max_bitrate: 320
threads: 1
# don't mind if tracks are missing from an album
match:
distance_weights:
missing_tracks: 0
paths:
default: $albumartist/$album%aunique{}/$track - $title
singleton: Non-Album/$artist - $title
comp: Compilations/$album%aunique{}/$track - $title
albumtype_soundtrack: Soundtracks/$album/$track $title
import:
write: yes
# we copy so pymix can remove the import src dir when the
# process has completed successfully.
copy: yes
move: no
resume: ask
incremental: no
incremental_skip_later: no
# if duplicates found then merge them in. This can happen
# if tracks for a compilation are split over multiple
# directories. Then each time beets tries to import one of
# the directories it will think that it is importing a
# duplicate compilation album. This is acceptable since the
# tracks should be distinct so we just ask beets to merge.
duplicate_action: merge
quiet_fallback: asis
timid: no
log: /config/beet.log
#lastgenre:
# auto: yes
# source: album
embedart:
auto: yes
fetchart:
auto: yes
replaygain:
auto: no
scrub:
auto: yes
replace:
'^\.': _
'[\x00-\x1f]': _
'[<>:"\?\*\|]': _
'[\xE8-\xEB]': e
'[\xEC-\xEF]': i
'[\xE2-\xE6]': a
'[\xF2-\xF6]': o
'[\xF8]': o
'\.$': _
'\s+$': ''
web:
host: 0.0.0.0
port: 8337
```
Thanks
Luke Purnell
Contributor guide
Assessment
This issue has not been assessed yet.