convert plugin crashed when filename has some unicode character (♥) UnicodeEncodeError
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 2.1k
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 31
Description
### Problem
This is the filename
`beet ls title:マジヤバもーそうLOVE` -f '$path'
```sh
D:\MusicLibrary\AnimeSongs\マジヤバもーそうLOVE♥ (オリジナル・バーシオン) - 合田彩, 柏山奈々美, 積田かよ子, 村井理沙子, 福原由莉奈, 小松真奈, 村上まどか & 月宮みど.flac
```
when use query to specific this file to convert ,cause error:
`UnicodeEncodeError: 'gbk' codec can't encode character '\u2665' in position 39: illegal multibyte sequence`
My system language is **Chinese**, it seems like the convert plugin encode the command string to **gbk**
enter the unicode string in ipython i figured out '\u2665' = `♥`
but the unicode character`♥` can't be find in gbk ,
i think it's no need to encode gbk on my system,so i edit the py file to return utf-8 directly, then it work properly
`C:\users\caichengtao05\appdata\local\programs\python\python37-32\lib\site-packages\beets\util\__init__.py` line 321
```python
def arg_encoding():
"""Get the encoding for command-line arguments (and other OS
locale-sensitive strings).
"""
try:
#======================== return utf-8 directly
# return locale.getdefaultlocale()[1] or 'utf-8'
return 'utf-8'
except ValueError:
# Invalid locale environment variable setting. To avoid
# failing entirely for no good reason, assume UTF-8.
return 'utf-8'
```
convert succeed:
```sh
C:\Users\caichengtao05>beet convert -d E:\MusicLibrary title:マジヤバもーそうLOVE
合田彩, 柏山奈々美, 積田かよ子, 村井理沙子, 福原由莉奈, 小松真奈, 村上まどか & 月宮みどり - マジヤバもーそうLOVE - マジヤバもーそうLOVE♥ (オリジナル・バーシオン)
Convert? (Y/n) y
convert: Encoding D:\MusicLibrary\AnimeSongs\マジヤバもーそうLOVE♥ (オリジナル・バーシオン) - 合田彩, 柏山奈々美, 積田
かよ子, 村井理沙子, 福原由莉奈, 小松真奈, 村上まどか & 月宮みど.flac
convert: Finished encoding D:\MusicLibrary\AnimeSongs\マジヤバもーそうLOVE♥ (オリジナル・バーシオン) - 合田彩, 柏山奈々美, 積田かよ子, 村井理沙子, 福原由莉奈, 小松真奈, 村上まどか & 月宮みど.flac
```
Running this command in verbose (`-vv`) mode:
```sh
$ beet -vv convert -d E:\MusicLibrary -p title:マジヤバもーそうLOVE
```
Led to this problem:
```
user configuration: C:\Users\caichengtao05\AppData\Roaming\beets\config.yaml
data directory: C:\Users\caichengtao05\AppData\Roaming\beets
plugin paths:
Sending event: pluginload
lyrics: Disabling google source: no API key configured.
inline: adding item field has_date
library database: D:\MusicLibrary\musiclibrary.db
library directory: D:\MusicLibrary
Sending event: library_opened
Traceback (most recent call last):
File "C:\Users\caichengtao05\AppData\Local\Programs\Python\Python37-32\Scripts\beet-script.py", line 11, in
load_entry_point('beets==1.4.9', 'console_scripts', 'beet')()
File "c:\users\caichengtao05\appdata\local\programs\python\python37-32\lib\site-packages\beets\ui\__init__.py", line 1266, in main
_raw_main(args)
File "c:\users\caichengtao05\appdata\local\programs\python\python37-32\lib\site-packages\beets\ui\__init__.py", line 1253, in _raw_main
subcommand.func(lib, suboptions, subargs)
File "c:\users\caichengtao05\appdata\local\programs\python\python37-32\lib\site-packages\beetsplug\convert.py", line 458, in convert_func
pipe.run_parallel()
File "c:\users\caichengtao05\appdata\local\programs\python\python37-32\lib\site-packages\beets\util\pipeline.py", line 445, in run_parallel
six.reraise(exc_info[0], exc_info[1], exc_info[2])
File "C:\Users\caichengtao05\AppData\Roaming\Python\Python37\site-packages\six.py", line 693, in reraise
raise value
File "c:\users\caichengtao05\appdata\local\programs\python\python37-32\lib\site-packages\beets\util\pipeline.py", line 358, in run
self.coro.send(msg)
File "c:\users\caichengtao05\appdata\local\programs\python\python37-32\lib\site-packages\beetsplug\convert.py", line 303, in convert_item
self.encode(command, original, converted, pretend)
File "c:\users\caichengtao05\appdata\local\programs\python\python37-32\lib\site-packages\beetsplug\convert.py", line 223, in encode
encode_cmd.append(args[i].encode(util.arg_encoding()))
UnicodeEncodeError: 'gbk' codec can't encode character '\u2665' in position 39: illegal multibyte sequence
```
Here's a link to the music files that trigger the bug (if relevant):
### Setup
* OS: win10
* Python version: 3.7.2
* beets version: 1.4.9
* Turning off plugins made problem go away (yes/no): yes
My configuration (output of `beet config`) is:
```yaml
lyrics:
bing_lang_from: []
auto: no
bing_client_secret: REDACTED
bing_lang_to:
google_API_key: REDACTED
google_engine_ID: REDACTED
genius_api_key: REDACTED
fallback:
force: no
local: no
sources:
- google
- lyricwiki
- musixmatch
- genius
library: D:/MusicLibrary/musiclibrary.db
directory: D:/MusicLibrary
import:
write: yes
copy: yes
move: no
link: no
hardlink: no
delete: no
resume: ask
incremental: no
incremental_skip_later: no
from_scratch: no
quiet_fallback: skip
none_rec_action: ask
timid: no
log: musiclibrary.log
autotag: no
quiet: no
singletons: yes
default_action: apply
languages: []
detail: yes
flat: no
group_albums: no
pretend: no
search_ids: []
duplicate_action: ask
bell: no
set_fields: {}
clutter: [Thumbs.DB, .DS_Store]
ignore:
- .*
- '*~'
- System Volume Information
- lost+found
ignore_hidden: yes
replace:
'[\\/]': _
^\.: _
'[\x00-\x1f]': _
'[<>:"\?\*\|]': _
\.$: _
\s+$: ''
^\s+: ''
^-: _
path_sep_replace: _
asciify_paths: no
art_filename: cover
max_filename_length: 0
aunique:
keys: albumartist album
disambiguators: albumtype year label catalognum albumdisambig releasegroupdisambig
bracket: '[]'
overwrite_null:
album: []
track: []
plugins:
- chroma
- convert
- fetchart
- fromfilename
- inline
- lastgenre
- lyrics
- web
pluginpath: []
threaded: yes
timeout: 5.0
per_disc_numbering: no
verbose: 0
terminal_encoding:
original_date: no
artist_credit: no
id3v23: no
va_name: Various Artists
chroma:
auto: yes
acoustid:
apikey: REDACTED
convert:
dest: none
format: aac
formats:
aac:
command: qaac64 --rate keep -v320 -q2 --copy-artwork -o $dest $source
extension: m4a
wav: ffmpeg -i $source -y -acodec pcm_s16le $dest
alac:
command: ffmpeg -i $source -y -vn -acodec alac $dest
extension: m4a
flac: ffmpeg -i $source -y -vn -acodec flac $dest
mp3: ffmpeg -i $source -y -vn -aq 2 $dest
opus: ffmpeg -i $source -y -vn -acodec libopus -ab 96k $dest
ogg: ffmpeg -i $source -y -vn -acodec libvorbis -aq 3 $dest
wma: ffmpeg -i $source -y -vn -acodec wmav2 -vn $dest
never_convert_lossy_files: yes
auto: no
tmpdir: none
copy_album_art: no
embed: yes
id3v23: inherit
quiet: no
pretend: no
threads: 4
max_bitrate: 500
paths: {}
no_convert: ''
album_art_maxwidth: 0
ui:
terminal_width: 80
length_diff_thresh: 10.0
color: yes
colors:
text_success: green
text_warning: yellow
text_error: red
text_highlight: red
text_highlight_minor: lightgray
action_default: turquoise
action: blue
format_item: $artist - $album - $title
format_album: $albumartist - $album
time_format: '%Y-%m-%d %H:%M:%S'
format_raw_length: no
sort_album: albumartist+ album+
sort_item: artist+ album+ disc+ track+
sort_case_insensitive: yes
item_fields:
has_date: 1 if len(str(year))==6 else 0
paths:
playlist:AnimeSongs: AnimeSongs/$title%if{$artist, - $artist,}
playlist:GalgameSongs: GalgameSongs/$title%if{$artist, - $artist,}
playlist:AnimeBgm: AnimeBgm/$title
playlist:GalgameBgm: GalgameBgm/$title
playlist:VocaloidBgm: VocaloidBgm/$title
playlist:ChineseSongs: ChineseSongs/$title%if{$artist, - $artist,}
playlist:EnglishSongs: EnglishSongs/$title%if{$artist, - $artist,}
playlist:JapaneseSongs: JapaneseSongs/$title%if{$artist, - $artist,}
playlist:KoreanSongs: KoreanSongs/$title%if{$artist, - $artist,}
playlist:VocaloidCN: VocaloidCN/$title%if{$artist, - $artist,}
playlist:VocaloidJP: VocaloidJP/$title%if{$artist, - $artist,}
playlist:AbsoluteMusic: AbsoluteMusic/$title%if{$artist, - $artist,}
playlist:Karaoke: Karaoke/$title - $length
playlist:Vtuber: Vtuber/$vtubername/$album/$title%if{$has_date,[$year],}
playlist:singer: Singer/$singername/$album/$title
playlist:LoveLive: LoveLive/$llgroupname/$album/$title
default: Album/$album%aunique{}/$track $title
singleton: singleton/$title
comp: Compilations/$album%aunique{}/$track $title
statefile: state.pickle
musicbrainz:
host: musicbrainz.org
ratelimit: 1
ratelimit_interval: 1.0
searchlimit: 5
match:
strong_rec_thresh: 0.04
medium_rec_thresh: 0.25
rec_gap_thresh: 0.25
max_rec:
missing_tracks: medium
unmatched_tracks: medium
distance_weights:
source: 2.0
artist: 3.0
album: 3.0
media: 1.0
mediums: 1.0
year: 1.0
country: 0.5
label: 0.5
catalognum: 0.5
albumdisambig: 0.5
album_id: 5.0
tracks: 2.0
missing_tracks: 0.9
unmatched_tracks: 0.6
track_title: 3.0
track_artist: 2.0
track_index: 1.0
track_length: 2.0
track_id: 5.0
preferred:
countries: []
media: []
original_year: no
ignored: []
required: []
ignored_media: []
ignore_data_tracks: yes
ignore_video_tracks: yes
track_length_grace: 10
track_length_max: 30
fetchart:
auto: yes
minwidth: 0
maxwidth: 0
enforce_ratio: no
cautious: no
cover_names:
- cover
- front
- art
- album
- folder
sources:
- filesystem
- coverart
- itunes
- amazon
- albumart
google_key: REDACTED
google_engine: 001442825323518660753:hrh5ch1gjzm
fanarttv_key: REDACTED
store_source: no
lastgenre:
whitelist: yes
min_weight: 10
count: 1
fallback:
canonical: no
source: album
force: yes
auto: yes
separator: ', '
prefer_specific: no
web:
host: 127.0.0.1
port: 8337
cors: ''
cors_supports_credentials: no
reverse_proxy: no
include_paths: no
pathfields: {}
album_fields: {}
```
Contributor guide
Assessment
This issue has not been assessed yet.