playlist: Address Unicode normalization differences when constructing queries
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 2.1k
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 31
Description
### Problem
Path in m3u files containing unicode characters do not generate proper queries, and thus the target files are never found. Given a file path like:
```
/Volumes/Music/Music/_organized/Gregor Cürten & Anselm Rogmans/Planes [Entr’acte]/02-Planes II.flac
```
I see (via some print statements) that this path is converted to this before querying the database:
```
b'/Volumes/Music/Music/_organized/Gregor C\xc3\xbcrten & Anselm Rogmans/Planes [Entr\xe2\x80\x99acte]/02-Planes II.flac'
```
This bytestream path does not return any results, though this path is queryable via other commands like `beet info` and the displayed path via things like `ls -f '$path'` matches.
It looks like this may be related to `PlaylistQuery:match` being slightly invalid? If I populate my original playlist with (1) the original path as it was in the playlist, and (2) the path copy-pasted from the output of a `beet info` command, I see the following paths being queries:
```
b'/Volumes/Music/Music/_organized/Gregor C\xc3\xbcrten & Anselm Rogmans/Planes [Entr\xe2\x80\x99acte]/02-Planes II.flac'
b'/Volumes/Music/Music/_organized/Gregor Cu\xcc\x88rten & Anselm Rogmans/Planes [Entr\xe2\x80\x99acte]/02-Planes II.flac'
```
If I read each of these as unicode strings (e.g. not as bytestreams) in Python, they are `==`, but of course as bytestreams they are not.
AFAICT these two strings are equivalent unicode representations, so probably the fix here is to simply normalize the paths in the m3u before querying. Experimentally, this seems to resolve the problem for several of the cases I'm seeing:
```
line = unicodedata.normalize('NFD', line)
```
But I don't know enough to say whether this is correct - possibly normalization like this should be part of the query pipeline in a more generalized way, rather than requiring it to be added ad hoc to every plugin?
Contributor guide
Research direction
Start with the playlist query path, especially PlaylistQuery:match, and reproduce the issue using an m3u entry containing composed and decomposed Unicode paths. Compare where the path becomes a byte string and determine the appropriate normalization boundary. Done means equivalent Unicode path representations produce matching database queries without breaking existing playlist lookups.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100