uutils / uutils/coreutils

Error: Windows stdio in console mode does not support writing non-UTF-8 byte sequences

Open
#8,523 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

P - Windows U - head U - tail
Dominant language
Rust
Stars
24.1k
Forks
2k
Avg merge
1d 5h
Merged PRs (30d)
365

Description

Issue:

It seems 0.10 coreutils head/tail doesn't gracefully handle outputting UTF-16 content in Windows console. This is somewhat annoying since in PowerShell (versions prior to PowerShell 6.0), the redirection operators (>/>>) default to writing files in UTF-16LE.

Error: head.exe: error writing 'standard output': Windows stdio in console mode does not support writing non-UTF-8 byte sequences

PS C:\Users\User> echo "foobar" > test
PS C:\Users\User> cat test
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: test   <UTF-16LE>
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ foobar
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────
PS C:\Users\User> head test
C:\Users\User/.dotfiles/opt/bin\head.exe: error writing 'standard output': Windows stdio in console mode does not support writing non-UTF-8 byte sequences

PS C:\Users\User> tail test

thread 'main' panicked at src\uu\tail\src\tail.rs:431:33:
called `Result::unwrap()` on an `Err` value: Error { kind: InvalidData, message: "Windows stdio in console mode does not support writing non-UTF-8 byte sequences" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

PS C:\Users\User> echo $Host
Name             : ConsoleHost
Version          : 5.1.26100.4768
InstanceId       : 1c7962ab-05f7-417d-a559-f5dd21650b9a
UI               : System.Management.Automation.Internal.Host.InternalHostUserInterface
CurrentCulture   : en-US
CurrentUICulture : en-US
PrivateData      : Microsoft.PowerShell.ConsoleHost+ConsoleColorProxy
DebuggerEnabled  : True
IsRunspacePushed : False
Runspace         : System.Management.Automation.Runspaces.LocalRunspace

--==--
Testing with https://github.com/JavaScriptDude/cygtail (Cygwin) looks like it has gracefully output.

PS C:\Users\User\Desktop\cygtail-20220415> echo "foobar" > test
PS C:\Users\User\Desktop\cygtail-20220415> ./tail.exe test
□□foobar

Anticipated Result:

head/tail gracefully process files created by simple redirection output (e.g. echo "foobar" > test). Not sure if that means inserted unknown character glyphs or similar or converting on the fly to work within the presumed UTF-8 limit of windows console.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the Windows PowerShell 5.1 case by creating a UTF-16LE file with redirection, then inspect the tail path at src/uu/tail/src/tail.rs:431 and the corresponding head implementation. Compare both commands' handling of Windows console output; done means head and tail process these files without the reported encoding error or panic, with behavior matching the issue's anticipated result.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.