python / python/cpython

iconv codecs corrupt stateful encodings when decoding incrementally

Open
#154,859 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

3.16 stdlib type-bug
Dominant language
Python
Stars
77.2k
Forks
36k
PR merge metrics
PR metrics pending

Description

Bug description:

The iconv-backed codecs start a fresh conversion on every call, so an incremental decode of a stateful encoding loses the shift state and silently returns wrong text. iso-2022-cn has no built-in codec, so the plain name reaches the iconv codec:

import codecs
blob = 'ABC中文DEF'.encode('iso-2022-cn')
print('one-shot:   ', blob.decode('iso-2022-cn'))
d = codecs.getincrementaldecoder('iso-2022-cn')()
print('incremental:', ''.join(d.decode(bytes([b])) for b in blob) + d.decode(b'', True))
one-shot:    ABC中文DEF
incremental: ABCVPNDDEF

codecs.iterdecode() and StreamReader.read(1) are wrong in the same way. A stateful encoding forced with the iconv: prefix is also affected, even when it has a built-in codec.

Either the codec objects need to keep one conversion across calls, or the iconv search function should refuse stateful encodings, which would make iso-2022-cn a LookupError.

The iconv codecs are new in 3.16 (gh-152997), so no released version is affected.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs
  • gh-154862

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the incremental-decoding example for the iconv-backed codec, then inspect the incremental decoder, codecs.iterdecode(), and StreamReader.read(1) paths described in the report. Done means stateful encodings preserve conversion state across calls, or the search function rejects unsupported stateful encodings consistently, with the shown cases covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.