beetbox / beetbox/beets

playlist: Address Unicode normalization differences when constructing queries

Open
#4,739 6 comments 0 reactions 0 assignees View on GitHub
bug playlist
Dominant language
Python
Stars
15.7k
Forks
2.1k
Avg merge
4d 21h
Merged PRs (30d)
31

Description

### Problem

Path in m3u files containing unicode characters do not generate proper queries, and thus the target files are never found. Given a file path like:
```
/Volumes/Music/Music/_organized/Gregor Cürten & Anselm Rogmans/Planes [Entr’acte]/02-Planes II.flac
```
I see (via some print statements) that this path is converted to this before querying the database:
```
b'/Volumes/Music/Music/_organized/Gregor C\xc3\xbcrten & Anselm Rogmans/Planes [Entr\xe2\x80\x99acte]/02-Planes II.flac'
```

This bytestream path does not return any results, though this path is queryable via other commands like `beet info` and the displayed path via things like `ls -f '$path'` matches.

It looks like this may be related to `PlaylistQuery:match` being slightly invalid? If I populate my original playlist with (1) the original path as it was in the playlist, and (2) the path copy-pasted from the output of a `beet info` command, I see the following paths being queries:

```
b'/Volumes/Music/Music/_organized/Gregor C\xc3\xbcrten & Anselm Rogmans/Planes [Entr\xe2\x80\x99acte]/02-Planes II.flac'
b'/Volumes/Music/Music/_organized/Gregor Cu\xcc\x88rten & Anselm Rogmans/Planes [Entr\xe2\x80\x99acte]/02-Planes II.flac'
```
If I read each of these as unicode strings (e.g. not as bytestreams) in Python, they are `==`, but of course as bytestreams they are not.

AFAICT these two strings are equivalent unicode representations, so probably the fix here is to simply normalize the paths in the m3u before querying. Experimentally, this seems to resolve the problem for several of the cases I'm seeing:
```
line = unicodedata.normalize('NFD', line)
```

But I don't know enough to say whether this is correct - possibly normalization like this should be part of the query pipeline in a more generalized way, rather than requiring it to be added ad hoc to every plugin?

Contributor guide

Open the contributing guide

Research direction

Start with the playlist query path, especially PlaylistQuery:match, and reproduce the issue using an m3u entry containing composed and decomposed Unicode paths. Compare where the path becomes a byte string and determine the appropriate normalization boundary. Done means equivalent Unicode path representations produce matching database queries without breaking existing playlist lookups.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.