xiph / xiph/flac

Windows problem with SetConsoleOutputCP CP_UTF8 and chcp 65001

Open
#894 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
2.4k
Forks
363
PR merge metrics
No merged PRs in 30d

Description

Some background:
https://hydrogenaudio.org/index.php/topic,128969.0.html
https://hydrogenaudio.org/index.php/topic,129063.0.html

Which seems to match the problems reported here:
https://stackoverflow.com/questions/39736901/chcp-65001-codepage-results-in-program-termination-without-any-error
https://superuser.com/questions/1118106/can-the-utf-8-code-page-identifier-65001-be-different-on-other-computers
https://stackoverflow.com/questions/878972/windows-cmd-encoding-change-causes-python-crash

So to summarize: SetConsoleOutputCP with CP_UTF8 or chcp 65001 only works on Windows 8 and later, and is likely to cause crashes in Windows 7 and earlier.
But still it seems bugs appear in Windows 10; output might be duplicated due to WriteFile() returns the number of UTF-16 characters written, not UTF-8 bytes.

The conclusion then is to only use the W APIs directly on Windows; WriteConsoleW() with UTF-16 if writing to console, and UTF-8 if redirected to file. This should work the same on all Windows versions (or at least since Windows XP).

For writing binary, WriteFile() should be used instead of WriteConsoleW(). But writing binary to the console doesn't make sense on Windows anyway, best to prevent this in some way?

Care must be taken to make sure piping works as expected, for example:
metaflac --export-tags-to=- file1.flac | metaflac --import-tags-from=- flac2.flac

Might be worth testing tags with broken UTF-8 also, as I suspect this will likely have problems in Windows due to being converted to UTF-16; if it works at all, it will be a lossy operation, which the user might find surprising, especially if the goal is to copy tags.

But also with binary output/input:
metaflac --list --data-format=binary file1.flac | metaflac --append --data-format=binary file2.flac

Due to limitations on PowerShell, it seems binary can't work due to it being UTF-16 only, and so it would be best to implement something like --data-format=hex.

I can do some testing on Windows XP in due time, and will post my findings here on what works and what doesn't.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no repository files, tests, or entry points. Start by reproducing the listed text and binary piping commands on Windows versions and compare console, redirected-file, and piped output; done means UTF-8 text and binary data work without crashes, duplication, or lossy conversion.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
cli, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.