`wc -m` returns character count instead of byte count in C/POSIX locale
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 24.1k
- Forks
- 2k
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 365
Description
Component
wc -m
Description
GNU wc checks MB_CUR_MAX to determine whether to count bytes or multibyte characters. When MB_CUR_MAX == 1 (C/POSIX locale), it treats each byte as a character.
static bool
wc (int fd, char const *file_x, struct fstatus *fstatus)
{
// [...]
/* If in the current locale, chars are equivalent to bytes, we prefer
counting bytes, because that's easier. */
! if (MB_CUR_MAX > 1)
{
count_bytes = print_bytes;
count_chars = print_chars;
}
else
{
count_bytes = print_bytes || print_chars;
count_chars = false;
}
However, uutils wc ignores locale and always counts UTF-8 characters using bytecount::num_chars().
pub(crate) fn count_bytes_chars_and_lines_fast<
// [...]
>(
handle: &mut R,
) -> (WordCount, Option<io::Error>) {
let mut total = WordCount::default();
let buf: &mut [u8] = &mut AlignedBuffer::default().data;
loop {
match handle.read(buf) {
Ok(0) => return (total, None),
Ok(n) => {
if COUNT_BYTES {
total.bytes += n;
}
if COUNT_CHARS {
! total.chars += bytecount::num_chars(&buf[..n]);
}
if COUNT_LINES {
total.lines += bytecount::count(&buf[..n], b'\n');
}
}
Err(ref e) if e.kind() == ErrorKind::Interrupted => (),
Err(e) => return (total, Some(e)),
}
}
}
Test / Reproduction Steps
$ echo -n "한글"|LC_ALL=C wc -m
6
$ echo -n "한글"|LC_ALL=C coreutils wc -m
2
Impact
wc -m produces different output than GNU in C/POSIX locale environments, breaking compatibility for scripts and CI pipelines that rely on locale-dependent character counting.
Recommendations
Check MB_CUR_MAX (or equivalent locale detection in rust) before counting characters. If MB_CUR_MAX == 1, return byte count instead of UTF-8 character count to match GNU behavior.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run the supplied LC_ALL=C reproduction and compare uutils wc with GNU coreutils. Start in src/uu/wc/src/count_fast.rs, then inspect the surrounding wc counting entry points for locale handling. Done means wc -m returns byte counts in the C/POSIX locale while preserving character counting in multibyte locales.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100