convert: Avoid an encoding error on Windows when in a non-Unicode locale
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 2.1k
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 31
Description
### Problem
Arg_encoding leads to a non-Unicode encoding being used. The paths need to be unicode to make Japanese characters, so they fail.
https://github.com/beetbox/beets/blob/master/beetsplug/convert.py#L222
```python
for i, arg in enumerate(args):
args[i] = Template(arg).safe_substitute({
'source': source,
'dest': dest,
})
if six.PY2:
encode_cmd.append(args[i])
else:
encode_cmd.append(args[i].encode(util.arg_encoding()))
```
https://github.com/beetbox/beets/blob/master/beets/util/__init__.py#L319
When I hardcode the ` try:` block to "utf-8", the problem is fixed.
2019-10-28 edit: Hardcode `arg_encoding` to always return `'utf-8'`.
https://github.com/beetbox/beets/blob/1b187fbf5345727e0dfdaea958a714f19e917a4e/beets/util/__init__.py#L328
I haven't set my default codepage to UTF8, but the process has [side-effects](https://superuser.com/questions/269818/change-default-code-page-of-windows-console-to-utf-8) on Windows so I don't know if that's what users should be expected to do.
My terminal can print special characters just fine, and beets has been able to write paths with special characters as well. This leads me to conclude that calling `arg_encoding()` isn't the right move here.
Running this command in verbose (`-vv`) mode:
```sh
λ beet -vv convert --format mp3 -y albumartist:2814
[snip]
convert: Encoding e:\Music\2814 - 2015 - 新しい日の誕生 (2016)\01. 恢复.flac
convert: Encoding e:\Music\2814 - 2015 - 新しい日の誕生 (2016)\02. 遠くの愛好家.flac
the: "2814" -> "2814"
convert: Encoding e:\Music\2814 - 2015 - 新しい日の誕生 (2016)\03. 新宿ゴールデン街.flac
convert: Encoding e:\Music\2814 - 2015 - 新しい日の誕生 (2016)\04. ふわっと.flac
Traceback (most recent call last):
File "\AppData\Local\Programs\Python\Python37\Scripts\beet-script.py", line 11, in
load_entry_point('beets==1.4.8', 'console_scripts', 'beet')()
File "c:\apps\cmder_mini\src\beets\beets\ui\__init__.py", line 1262, in main
_raw_main(args)
File "c:\apps\cmder_mini\src\beets\beets\ui\__init__.py", line 1249, in _raw_main
subcommand.func(lib, suboptions, subargs)
File "c:\apps\cmder_mini\src\beets\beetsplug\convert.py", line 453, in convert_func
pipe.run_parallel()
File "c:\apps\cmder_mini\src\beets\beets\util\pipeline.py", line 445, in run_parallel
six.reraise(exc_info[0], exc_info[1], exc_info[2])
File "\AppData\Local\Programs\Python\Python37\lib\site-packages\six.py", line 693, in reraise
raise value
File "c:\apps\cmder_mini\src\beets\beets\util\pipeline.py", line 358, in run
self.coro.send(msg)
File "c:\apps\cmder_mini\src\beets\beetsplug\convert.py", line 302, in convert_item
self.encode(command, original, converted, pretend)
File "c:\apps\cmder_mini\src\beets\beetsplug\convert.py", line 222, in encode
encode_cmd.append(args[i].encode(util.arg_encoding()))
File "\AppData\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 23-29: character maps to
```
### Setup
* OS: Win10 x64
* Python version: 3.7.1
* beets version: 1.4.8 (dev)
* Terminal: Cmder, cmd, powershell
Contributor guide
Research direction
Start in beetsplug/convert.py at encode(), especially line 222, and trace util.arg_encoding() in beets/util/__init__.py around line 319. Reproduce the reported Windows non-Unicode-locale failure with beet -vv convert and Unicode paths. Done means conversion no longer raises UnicodeEncodeError for those paths without requiring a system code-page change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100