UnicodeEncodeError fallback discards all non-ASCII output, including characters the target encoding supports

Summary

When output cannot be encoded for the destination stream, OutputProducer.out falls back to re-encoding the whole document as ASCII with errors='ignore'. Because the fallback targets ASCII rather than the stream's actual encoding, a single unrepresentable character anywhere in the output causes every non-ASCII character in the entire document to be discarded, including characters the destination encoding represents perfectly well.

On a Windows console in a non-UTF-8 locale this means one emoji, CJK character or mathematical symbol anywhere in a result set silently strips every accented Latin character from the rest of it. For anyone working in Danish, German, French, Spanish or any other language with accented characters, correct data is replaced with plausible-looking wrong data, and the only signal is a warning on stderr while the corrupted document goes to stdout.

Reproduction

Self-contained, no Azure or network required:

import io
from knack.cli import CLI
from knack.output import OutputProducer
from knack.util import CommandResultItem

def emit(payload, label):
    # A stream whose encoding cannot represent every character, as on a Windows console in a non-UTF-8 locale.
    buf = io.TextIOWrapper(io.BytesIO(), encoding="cp1252", errors="strict")
    producer = OutputProducer(cli_ctx=CLI())
    producer.out(CommandResultItem(payload), formatter=producer.get_formatter("json"), out_file=buf)
    buf.flush()
    print(f"{label}\n  in : {payload}\n  out: {buf.buffer.getvalue().decode('cp1252').strip()}\n")

# Every character here is representable in cp1252.
emit({"text": "ae[æ] oe[ø] aa[å] dash[—]"}, "A. all characters representable in cp1252")

# Identical, plus one character that is NOT representable (U+221E INFINITY).
emit({"text": "ae[æ] oe[ø] aa[å] dash[—] inf[∞]"}, "B. same, plus one unrepresentable character")

Output on stdout (case B additionally logs Unable to encode the output with cp1252 encoding. Unsupported characters are discarded. to stderr):

A. all characters representable in cp1252
  in : {'text': 'ae[æ] oe[ø] aa[å] dash[—]'}
  out: {
  "text": "ae[æ] oe[ø] aa[å] dash[—]"
}

B. same, plus one unrepresentable character
  in : {'text': 'ae[æ] oe[ø] aa[å] dash[—] inf[∞]'}
  out: {
  "text": "ae[] oe[] aa[] dash[] inf[]"
}

Case A shows that cp1252 handles æ, ø, å and the em dash without difficulty. Case B adds one character it cannot handle, and all four of those previously fine characters are discarded along with it.

The same behaviour appears through Azure CLI on a Windows console whose code page is not UTF-8, for any command whose output contains a character outside the console encoding:

WARNING: Unable to encode the output with cp1252 encoding. Unsupported characters are discarded.

Expected behaviour

Only characters that the destination encoding genuinely cannot represent should be affected, and ideally they should degrade to something visible rather than vanishing.

Cause

knack/output.py, lines 154 to 158 (identical in the released v0.14.0 tag and on the default dev branch, currently 52f769a):

except UnicodeEncodeError:
    logger.warning("Unable to encode the output with %s encoding. Unsupported characters are discarded.",
                   out_file.encoding)
    print(output.encode('ascii', 'ignore').decode('utf-8', 'ignore'),
          file=out_file, end='')

The warning reports out_file.encoding, but the retry hardcodes 'ascii'. The destination encoding is never attempted on the retry, so its capabilities are never used.

The warning text is inaccurate in two further ways. It names an encoding that the fallback does not use, and it says "Unsupported characters are discarded" when supported characters are discarded too.

Separately, .decode('utf-8', 'ignore') applied to bytes that were just produced by .encode('ascii', 'ignore') is a no-op round trip, since ASCII output is always valid UTF-8.

Suggested fix

Encode to the stream's own encoding, and use an error handler that preserves what it can:

except UnicodeEncodeError:
    # Retry with the stream's own encoding so that characters it *can* represent survive.
    # Encoding to 'ascii' here would discard every non-ASCII character in the document,
    # not just the ones the destination cannot represent.
    encoding = out_file.encoding or 'ascii'
    logger.warning("Unable to encode some characters with %s encoding. "
                   "They are replaced with '?'.", encoding)
    print(output.encode(encoding, 'replace').decode(encoding),
          file=out_file, end='')

Comparison on the case B payload:

original                  : ae[æ] oe[ø] aa[å] dash[—] inf[∞]
current (ascii + ignore)  : ae[] oe[] aa[] dash[] inf[]
cp1252 + replace          : ae[æ] oe[ø] aa[å] dash[—] inf[?]
cp1252 + backslashreplace : ae[æ] oe[ø] aa[å] dash[—] inf[\u221e]

Both replace and backslashreplace fix the bug reported here, since either one uses the destination encoding and therefore stops discarding characters it supports. They differ only in how the genuinely unrepresentable characters are rendered, which matters because this fallback is shared by every output format.

Why replace rather than backslashreplace

backslashreplace preserves more information, and for characters in the Basic Multilingual Plane it is the better of the two: it emits \uXXXX, which is also a valid JSON escape, so the document stays valid JSON and a parser reads the original character straight back. For JSON output that is lossless.

It breaks down outside the BMP. For emoji and other astral-plane characters Python emits \U0001f600, with a capital U and eight digits. JSON permits only \uXXXX, so a document containing one no longer parses:

>>> json.loads('{"t": "\U0001f600"}')
json.decoder.JSONDecodeError: Invalid \escape: line 1 column 8 (char 7)

That matters most in exactly the situation this issue is about. On Windows, redirecting to a file (az ... > out.json) also encodes with the locale codec, so a single emoji in a tag or a description would produce a file that cannot be parsed at all. Trading today's silent corruption for a hard parse failure did not seem like the right call for a code path shared by every command of every knack-based CLI, so the attached PR uses replace.

Making the non-BMP case JSON-correct would need a surrogate-pair encoder, which does not belong in a format-agnostic code path that also serves YAML, table and TSV output. If preserving the character identity is worth more than guaranteed parseability, switching the handler is a one-word change.

History, for context

The encode('ascii', 'ignore') fallback is long-standing and is present as far back as the v0.4.5 tag. Two later changes made it matter more than it used to:

• #115 (2018-10-30) set ensure_ascii=False on the JSON formatter, specifically so that non-ASCII characters would be preserved in output. That is what causes real non-ASCII characters to reach the console, and therefore what makes this fallback fire on Windows. The fallback undoes that PR's stated purpose whenever any single character in the document is unrepresentable.
• #178 (2020-03-10) added the warning shown above. The diff only added the two logger.warning lines, so the mismatch between the message and the ASCII fallback beneath it appears to be an oversight rather than a deliberate choice.

Impact

OutputProducer.out is the generic output path, so this affects every command of every knack-based CLI, including all of Azure CLI, on any Windows console that is not set to UTF-8. Azure CLI pins knack~=0.14.0, so a patch release in the 0.14.x series would reach Azure CLI users without any change on their side.

The failure mode is quiet. The warning goes to stderr, so tooling that captures stdout, including scripts and AI agents that parse JSON output, receives corrupted values with no indication that anything was lost. A missing letter inside a customer name, a file path or an exception message is not distinguishable from data that never had one.

Environment

knack        0.14.0
azure-cli    2.88.0
Python       3.14.5 (Windows)
Console      ACP 1252, OutputCP 437
OS           Windows 11

---

Disclosure: the investigation behind this issue, the reproduction script and this write-up were produced with Claude Code. The reproduction and the regression tests in the linked PR were executed against dev at 52f769a.

Start in knack/output.py around lines 154-158 and run the self-contained cp1252 reproduction from the issue. Update the fallback so supported characters survive while unrepresentable ones are visibly replaced, then add or run regression coverage for both representable and unsupported characters across the output path.