beetbox / beetbox/beets

Errors during (incremental) import lose all progress

Open
#5,904 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
15.7k
Forks
2.1k
Avg merge
4d 21h
Merged PRs (30d)
31

Description

### Problem

If an import is stopped by a crash or SIGINT, Beets appears to abandon all progress, forcing imports to be restarted.

Instead, import progress should be persisted at the file level: once Beets has finished processing a file's metadata, it should be saved to the database and flushed to disk, so that a crash will not lose that data.

### Context

I've been trying to import a reasonably sized library (~22k files) to Beets for about a week now and I've only succeeded in importing 170 tracks, despite Beets running for 15 hours continuously during its longest run.

I keep running into various network and API errors:
- #5903 (MusicBrainz)
- #5893
- #5898

When encountering one of these errors, Beets aborts entirely (#5903). After attempting to let Beets run overnight, I found that it crashed almost immediately after I fell asleep, wasting that time.

Adding to my frustration, **once I attempt to resume the import, Beets has lost all progress** and attempts to restart the import from the beginning. Thousands and thousands of API requests and metadata results are simply abandoned.

Related:
- #803
- #3139

### Setup

* OS: macOS Sequoia 15.5 (24F74)
* Python version: 3.13.5
* beets version: `master` as of 0fec858a13854dd16cba5fcd6e698da148f4672d (2.3.1), plus some patches:
* https://github.com/beetbox/beets/pull/5893
* https://github.com/beetbox/beets/pull/5898
* https://github.com/beetbox/beets/pull/5897
* Turning off plugins made problem go away (yes/no): N/A, maybe yes if Beets doesn't make network requests unless plugins are enabled?

My configuration (output of beet config) is:

```yaml
directory: ~/Music/beets

# --------------- Plugins ---------------

plugins: bandcamp chroma convert discogs duplicates embedart fetchart fromfilename lastgenre lyrics spotify

# --------------- UI ---------------

verbose: yes
logging:
format: '{asctime} {name} {levelname}: {message}'
levels:
beets.plugins: INFO
musicbrainzngs.mbxml: WARNING
lyrics:
auto: yes
sources: lrclib genius tekstowo
synced: yes
translate:
api_key: REDACTED
from_languages: []
to_language:
dist_thresh: 0.11
google_API_key: REDACTED
google_engine_ID: REDACTED
genius_api_key: REDACTED
fallback:
force: no
local: no
print: no

import:
move: yes
copy: no
duplicate_action: skip
log: ~/.cache/beets/beets.log
languages: [en]
group_albums: yes
musicbrainz:
external_ids:
discogs: yes
spotify: yes
bandcamp: yes
deezer: no
tidal: no
source_weight: 0.5
host: musicbrainz.org
https: no
ratelimit: 1
ratelimit_interval: 1
searchlimit: 5
genres: no
extra_tags: []
chroma:
auto: yes
source_weight: 0.5
convert:
delete_originals: yes
formats:
mp3: ffmpeg -y -i $source -c:a libmp3lame -q:a 0 -map_metadata 0 -map_metadata 0:s:0 $dest
aac:
command: ffmpeg -i $source -y -vn -acodec aac -aq 1 $dest
extension: m4a
alac:
command: ffmpeg -i $source -y -vn -acodec alac $dest
extension: m4a
flac: ffmpeg -i $source -y -vn -acodec flac $dest
opus: ffmpeg -i $source -y -vn -acodec libopus -ab 96k $dest
ogg: ffmpeg -i $source -y -vn -acodec libvorbis -aq 3 $dest
wma: ffmpeg -i $source -y -vn -acodec wmav2 -vn $dest
dest:
pretend: no
link: no
hardlink: no
threads: 12
format: mp3
id3v23: inherit
max_bitrate:
auto: no
auto_keep: no
tmpdir:
quiet: no
embed: yes
paths: {}
no_convert: ''
never_convert_lossy_files: no
copy_album_art: no
album_art_maxwidth: 0
playlist:
bandcamp:
source_weight: 0.5
art: yes
comments_separator: "\n\u2014\u2014\u2014\n"
genre:
capitalize: yes
mode: progressive
maximum: 0
always_include: []
include_digital_only_tracks: yes
search_max: 2
exclude_extra_fields: []
truncate_comments: no
fetchart:
cautious: yes
sources: itunes *
auto: yes
minwidth: 0
maxwidth: 0
quality: 0
max_filesize: 0
enforce_ratio: no
cover_names:
- cover
- front
- art
- album
- folder
store_source: no
high_resolution: no
deinterlace: no
cover_format:
google_key: REDACTED
google_engine: REDACTED
lastfm_key: REDACTED
fanarttv_key: REDACTED
duplicates:
album: yes
merge: yes
checksum: ''
copy: ''
count: no
delete: no
format: ''
full: no
keys: []
move: ''
path: no
tiebreak: {}
strict: no
tag: ''
remove: no
lastgenre:
canonical: yes
count: 5
force: yes
min_weight: 10
separator: '; '
whitelist: yes
fallback:
source: album
keep_existing: no
auto: yes
prefer_specific: no
title_case: yes
extended_debug: no
embedart:
maxwidth: 0
auto: yes
compare_threshold: 0
ifempty: no
remove_art_file: no
quality: 0
spotify:
source_weight: 0.5
mode: list
tiebreak: popularity
show_failures: no
artist_field: albumartist
album_field: album
track_field: title
region_filter:
regex: []
client_id: REDACTED
client_secret: REDACTED
tokenfile: spotify_token.json
search_query_ascii: no
discogs:
source_weight: 0.5
apikey: REDACTED
apisecret: REDACTED
tokenfile: discogs_token.json
user_token: REDACTED
separator: ', '
index_tracks: no
append_style_genre: no
search_limit: 5
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.