Error: Windows stdio in console mode does not support writing non-UTF-8 byte sequences
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 24.1k
- Forks
- 2k
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 365
Description
Issue:
It seems 0.10 coreutils head/tail doesn't gracefully handle outputting UTF-16 content in Windows console. This is somewhat annoying since in PowerShell (versions prior to PowerShell 6.0), the redirection operators (>/>>) default to writing files in UTF-16LE.
Error: head.exe: error writing 'standard output': Windows stdio in console mode does not support writing non-UTF-8 byte sequences
PS C:\Users\User> echo "foobar" > test
PS C:\Users\User> cat test
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ File: test <UTF-16LE>
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ foobar
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────
PS C:\Users\User> head test
C:\Users\User/.dotfiles/opt/bin\head.exe: error writing 'standard output': Windows stdio in console mode does not support writing non-UTF-8 byte sequences
PS C:\Users\User> tail test
thread 'main' panicked at src\uu\tail\src\tail.rs:431:33:
called `Result::unwrap()` on an `Err` value: Error { kind: InvalidData, message: "Windows stdio in console mode does not support writing non-UTF-8 byte sequences" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
PS C:\Users\User> echo $Host
Name : ConsoleHost
Version : 5.1.26100.4768
InstanceId : 1c7962ab-05f7-417d-a559-f5dd21650b9a
UI : System.Management.Automation.Internal.Host.InternalHostUserInterface
CurrentCulture : en-US
CurrentUICulture : en-US
PrivateData : Microsoft.PowerShell.ConsoleHost+ConsoleColorProxy
DebuggerEnabled : True
IsRunspacePushed : False
Runspace : System.Management.Automation.Runspaces.LocalRunspace
--==--
Testing with https://github.com/JavaScriptDude/cygtail (Cygwin) looks like it has gracefully output.
PS C:\Users\User\Desktop\cygtail-20220415> echo "foobar" > test
PS C:\Users\User\Desktop\cygtail-20220415> ./tail.exe test
□□foobar
Anticipated Result:
head/tail gracefully process files created by simple redirection output (e.g. echo "foobar" > test). Not sure if that means inserted unknown character glyphs or similar or converting on the fly to work within the presumed UTF-8 limit of windows console.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the Windows PowerShell 5.1 case by creating a UTF-16LE file with redirection, then inspect the tail path at src/uu/tail/src/tail.rs:431 and the corresponding head implementation. Compare both commands' handling of Windows console output; done means head and tail process these files without the reported encoding error or panic, with behavior matching the issue's anticipated result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100