base64: stdin buffering causes memory exhaustion and performance degradation

Running base64 --decode on stdin loads the entire input into memory before processing begins. This blocks on EOF and exhausts RAM with large streams.

Reproduction

Create a 500MB base64-encoded file:

dd if=/dev/zero bs=1M count=100 2>/dev/null | base64 > /tmp/large_base64.txt
# Creates ~133MB file (base64 expansion)

Test uutils:

head -c 500000000 /tmp/large_base64.txt | /usr/bin/time -l ./target/debug/coreutils base64 --decode > /dev/null 2>&1

Test GNU base64:

head -c 500000000 /tmp/large_base64.txt | /usr/bin/time -l base64 --decode > /dev/null 2>&1

Test Results:

Result: uutils uses ~331x more memory (325 MB vs 1 MB) on the same 500MB stream.

Root Cause

https://github.com/uutils/coreutils/blob/6cf5b06d8be0d8b6b4882dcdd1e7838376c6262d/src/uu/base32/src/base_common.rs#L166-L170

read_to_end() waits for EOF before returning. Files larger than RAM cause memory exhaustion.

What GNU does

GNU coreutils (basenc.c) uses a chunk-based loop:

No buffering, no EOF blocking, handles infinite streams.

References

1. GitHub Issue #8574: base64 almost 8x slower than GNU for large files

   • https://github.com/uutils/coreutils/issues/8574

2. Hacker News Discussion: Show HN: ut - Rust based CLI utilities

   • Main thread: https://news.ycombinator.com/item?id=45483531
   • Key comment (collinfunk): https://news.ycombinator.com/item?id=45485652
   • Topic: uutils reads stdin to EOF instead of streaming, exhausts RAM
   • Quote: "Note that uutils does not work if the file does not fit into memory... With uutils doing the same would exhaust your systems memory until either it freezes or oomd kills the process."

3. GNU coreutils Release Notes: Version 9.5 (Mar 28, 2024)

   • "base32 and base64 no longer require padding when decoding"
   • Indicates modern GNU behavior moving toward STANDARD_NO_PAD for stdin

Start with src/uu/base32/src/base_common.rs at lines 166-170, where read_to_end() is identified as the source of whole-input buffering. Compare the reported GNU basenc chunk-based behavior, then run the reproduction commands against uutils and GNU base64. Done means decoding large stdin streams without waiting for EOF or exhausting memory.