python / python/cpython

`utf-7-imap` incremental decoder and stream reader fail on split input

Open
#156,965 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stdlib type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

utf-7-imap registers an incremental decoder and a stream reader that call the one-shot decoder with no state, so a shift sequence split across chunks raises instead of being buffered. io.TextIOWrapper.read(n), codecs.iterdecode and codecs.StreamReader.read all fail on bytes the codec's own encoder produced.

import io

text = '台' * 5000
data = text.encode('utf-7-imap')
f = io.TextIOWrapper(io.BytesIO(data), encoding='utf-7-imap')
print(f.read(100) == text[:100])
UnicodeDecodeError: 'utf-7-imap' codec can't decode bytes in position 0-8191: unterminated shift sequence

Expected: True. Doc/library/codecs.rst says the joined output of calls to the incremental decode method is the same as decoding the joined input with the stateless decoder.

The incremental encoder breaks the same sentence more mildly: encoding '~peter/mail/台北/日本語' one character at a time joins to b'~peter/mail/&U,A-&Uxc-/&ZeU-&Zyw-&ip4-' where one shot gives b'~peter/mail/&U,BTFw-/&ZeVnLIqe-', though both decode back to the same text.

The codec is new in 3.16, so no released version is affected.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs
  • gh-156967

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the utf-7-imap incremental decoder and stream reader paths exercised by io.TextIOWrapper.read, codecs.iterdecode, and codecs.StreamReader.read. Read the incremental codec guidance in Doc/library/codecs.rst, then verify that split input works and that joined incremental results match the stateless decoder behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.