beetbox / beetbox/beets

Inline plugin - iterating over items for album_fields significantly slows down operations

Open
#5,773 10 comments 1 reaction 0 assignees Claimed by @snejus View on GitHub
inline
Dominant language
Python
Stars
15.7k
Forks
2.1k
Avg merge
4d 21h
Merged PRs (30d)
31

Description

### Problem

Using even a simple `album_field` definition such as:

```yaml
album_fields:
album_bitrate: |
total = 0
for item in items:
total += item.bitrate
```

**_significantly_** degrades performance.

Without:
```
root@6ea1b5cf64a4:/# time beet move -p 'title:Von dutch'
Moving 3 items.
[...]

real 0m2.702s
user 0m4.211s
sys 0m0.399s
```

With:
```
Moving 3 items.
[...]

real 0m15.705s
user 0m5.028s
sys 0m1.892s
```

With more items, it was significantly worse:

w/o:
```
root@6ea1b5cf64a4:/# time beet move -p 'artist:Charli XCX'
Moving 175 items.
[...]

real 0m5.264s
user 0m6.991s
sys 0m0.467s
```

with: (I actually gave up letting it run)
```
Moving 175 items.
[...]
^CTraceback (most recent call last):
File "/lsiopy/lib/python3.12/site-packages/beets/ui/__init__.py", line 1870, in main
_raw_main(args)
File "/lsiopy/lib/python3.12/site-packages/beets/ui/__init__.py", line 1849, in _raw_main
subcommand.func(lib, suboptions, subargs)
File "/lsiopy/lib/python3.12/site-packages/beets/ui/commands.py", line 2216, in move_func
move_items(
File "/lsiopy/lib/python3.12/site-packages/beets/ui/commands.py", line 2146, in move_items
objs = [o for o in objs if (isalbummoved if album else isitemmoved)(o)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/ui/commands.py", line 2141, in isitemmoved
return item.path != item.destination(basedir=dest)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 1115, in destination
subpath = self.evaluate_template(subpath_tmpl, True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 701, in evaluate_template
return t.substitute(
^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 559, in substitute
res = self.interpret(values, functions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 552, in interpret
return self.expr.evaluate(Environment(values, functions))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 240, in evaluate
out.append(part.evaluate(env))
^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 177, in evaluate
arg_vals = [expr.evaluate(env) for expr in self.args]
^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 240, in evaluate
out.append(part.evaluate(env))
^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 145, in evaluate
if self.ident in env.values:
^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 813, in __contains__
File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 489, in __getitem__
value = self._get(key)
^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 475, in _get
return self._get_formatted(self.album, key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 124, in _get_formatted
value = model._type(key).format(model.get(key))
^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 449, in _get
return getters[key](self)
^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beetsplug/inline.py", line 120, in _func_func
func.__globals__.update(_dict_for(obj))
^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beetsplug/inline.py", line 102, in _dict_for
out["items"] = list(obj.items())
^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 1295, in items
return self._db.items(dbcore.MatchQuery("album_id", self.id))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 1697, in items
return self._fetch(Item, query, sort or self.get_default_item_sort())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 1675, in _fetch
return super()._fetch(model_cls, query, sort)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 1257, in _fetch
rows = tx.query(sql, subvals)
^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 956, in query
cursor = self.db._connection().execute(statement, subvals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

real 5m15.560s
user 0m19.222s
sys 0m36.162s
```

### Setup

* OS: Unraid/Docker (linuxserver.io image)
* Python version: Python 3.12.10
* beets version: beets version 2.3.0
* Turning off plugins made problem go away (yes/no): Yes

My configuration (output of `beet config`) is:

```yaml
directory: /music/music
# --------------- Main ---------------

library: /config/library.db
pluginpath: /config/plugins

# --------------- Plugins ---------------

plugins:
- embedart
- fetchart
- info
- albumtypes
- lastgenre
- yearfixer
- fromfilename
- mbsync
- mbsubmit
- unimported
- zero
- missing
- inline
- badfiles
- substitute
- duplicates
- importreplace
- web
- originquery

# --------------- Performance ---------------

threaded: yes

# --------------- UI ---------------

verbose: no

# --------------- Tagging ---------------

per_disc_numbering: yes

import:
log: /config/beet.log
write: yes
copy: no
move: yes
resume: yes
incremental: yes
incremental_skip_later: yes
quiet: no
quiet_fallback: skip
from_scratch: no
default_action: apply
none_rec_action: ask
duplicate_action: ask
autotag: yes
art_filename: folder
mbcollection:
auto: yes
collection: XXXX
remove: yes
musicbrainz:
user: XXXX
pass: XXXX
extra_tags:
- year
- catalognum
- country
- media
- label
zero:
fields: comments
auto: yes
keep_fields: []
update_database: no

# --------------- Import ---------------

clutter:
- Thumbs.DB
- Thumbs.db
- .DS_Store
- '**.zip'
- '**.torrent'
- '**.ini'
- '**.txt'
- '**.nfo'
- '**.m3u'
missing:
format: $albumartist - $album - $title
count: no
total: no
album: no

match:
strong_rec_thresh: 0.08
max_rec:
missing_tracks: medium
unmatched_tracks: medium
preferred:
countries:
- US
- XE
- JP
- GB|UK
media: [Digital Media|File, CD, SACD]
original_year: no
item_fields:
is_mp3_aac: 1 if format == "MP3" or format == "AAC" else 0
is_flac: 1 if format == "FLAC" else 0
disc0: disc
disctotal0: disctotal
album_fields:
title0: title
albumtypes:
types:
- ep: 'EP - '
- single: 'Single - '
- live: 'Live - '
- remix: 'Remix - '
bracket: ''
ignore_va: [compilation]
paths:
default: '%substitute{$albumartist}/($original_year) $albumartist - %ifdef{altalbum,$altalbum,$album}%ifdef{transtitle, [$transtitle]} [%ifdef{disambig,$disambig - }%if{$atypes,$atypes}${format}%if{$is_mp3_aac, - $album_bitrate}%if{$is_flac, - $max_bitdepth - $max_samplerate}]/%if{$multidisc,Disc $disc0/}${track} - %ifdef{alttitle,$alttitle,$title}'
collection::^.+: _Other/$collection/($original_year) $albumartist - %ifdef{altalbum,$altalbum,$album}%ifdef{transtitle, [$transtitle]} [%ifdef{disambig,$disambig - }${format}%if{$is_mp3_aac, - $album_bitrate}%if{$is_flac, - $max_bitdepth - $max_samplerate}]/%if{$multidisc,Disc $disc0/}${track} - %ifdef{alttitle,$alttitle,$title}
substitute:
bladee &.*: Bladee
blu &.*: Blu
bob dylan &.*: Bob Dylan
broadcast and.*: Broadcast
casiopea.*: Casiopea
czarface.*: Czarface
"death\u2019s dynamic shroud and.*": "death\u2019s dynamic shroud"
dizzy gillespie.*: Dizzy Gillespie
drake &.*: Drake
"el\u2010p feat.*": "El\u2010P"
evaboy.*: miya lowe
fred again...*: Fred again..
.*gene clark with.*: Gene Clark
gorillaz feat.*|spacemonkeyz &.*: Gorillaz
"Hiromi\u2019s Sonicbloom|\u4E0A\u539F\u3072\u308D\u307F \u30B6\u30FB\u30C8\u30EA\u30AA\u30FB\u30D7\u30ED\u30B8\u30A7\u30AF\u30C8": Hiromi
iglooghost.*: Iglooghost
"jay\u2010z &.*": "JAY\u2010Z"
john williams.*: John Williams
jpegmafia.*: JPEGMAFIA
"kanye west.*|kids see ghosts|\\\xA5\\$": Kanye West
laufey with.*: Laufey
macroblank &.*: Macroblank
mf doom &.*|jj doom|dangerdoom|viktor vaughn|king geedorah|nehruviandoom|westsidedoom: MF DOOM
neil young &.*: Neil Young
quasimoto: Madlib
rita lee &.*: Rita Lee
round table.*: Round Table
ryusenkei.*: Ryusenkei
saint pepsi.*: SAINT PEPSI
"Stan Getz & Jo\xE3o Gilberto.*": "Stan Getz & Jo\xE3o Gilberto"
Sufjan Stevens &.*: Sufjan Stevens
thaiboy digital &.*: Thaiboy Digital
the moody blues.*: The Moody Blues
the velvet underground.*: The Velvet Underground
.*tommy heavenly.*: "Tommy february\u2076"
trent reznor & atticus ross.*: Trent Reznor & Atticus Ross
yung lean &.*: Yung Lean
^(.*?)( & Madlib| & The Alchemist).*: \1

importreplace:
replacements: [{item_fields: artist artist_credit artists artists_credit composer, album_fields: artist artist_credit artists artists_credit composer, replace: {'Ye(?!\w)': Kanye West, 'Charli xcx(?!\w)': Charli XCX, 'RYUSENKEI(?!\w)': Ryusenkei}}, {item_fields: artist_sort artists_sort composer_sort, album_fields: artist_sort artists_sort composer_sort, replace: {'Ye(?!\w)': 'West, Kanye', 'Charli xcx(?!\w)': Charli XCX, 'RYUSENKEI(?!\w)': Ryusenkei}}]
fetchart:
auto: no
cover_names: folder cover front
minwidth: 500
maxwidth: 3000
high_resolution: yes
enforce_ratio: 0.5%
sources:
- filesystem
- itunes
- amazon
- albumart
- coverart: releasegroup
deinterlace: yes
cover_format: JPEG
quality: 98
max_filesize: 0
cautious: no
store_source: no
google_key: REDACTED
google_engine: REDACTED
fanarttv_key: REDACTED
lastfm_key: REDACTED
embedart:
auto: no
maxwidth: 720
quality: 95
compare_threshold: 0
ifempty: no
remove_art_file: no
unimported:
ignore_extensions: jpg png zip jepg log cue yaml db
ignore_subdirectories: []
lastgenre:
whitelist: /config/whitelist.txt
auto: no
canonical: yes
count: 5
fallback: ''
force: yes
min_weight: 10
prefer_specific: no
source: album
title_case: yes
separator: '; '
keep_existing: no
extended_debug: no
originquery:
origin_file: origin.yaml
tag_patterns:
media: $.Media
year: $."Edition year"
label: $."Record label"
catalognum: $."Catalog number"
albumdisambig: $.Edition
web:
host: 0.0.0.0
port: 8337
cors: ''
cors_supports_credentials: no
reverse_proxy: no
include_paths: no
readonly: yes
duplicates:
album: no
checksum: ''
copy: ''
count: no
delete: no
format: ''
full: no
keys: []
merge: no
move: ''
path: no
tiebreak: {}
strict: no
tag: ''
pathfields: {}
yearfixer:
auto: no
force: no
mbsubmit:
format: $track. $title - $artist ($length)
threshold: medium
picard_path: picard

```

Contributor guide

Open the contributing guide

Research direction

Start with beetsplug/inline.py, especially _dict_for and _func_func, then follow the beet move path through beets/library.py and template evaluation in beets/dbcore/db.py. Reproduce the slowdown with the album_fields example and the provided beet move commands; done means the inline album-field case no longer causes the reported severe slowdown without changing its result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.