python / python/cpython

array: unpickling an odd-length 'w' array across endianness raises ValueError (wrong UTF-16/UTF-32 item size)

Open
#155,850 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

extension-modules type-bug
Dominant language
Python
Stars
77.2k
Forks
36k
PR merge metrics
PR metrics pending

Description

Bug report

array.array pickles carry a machine-format code so the receiver can decode
data from a different platform. The decoder's format table
(mformat_descriptors in Modules/arraymodule.c) stores a wrong item size
for the Unicode formats: UTF-16 entries say 4 bytes per item (actual: 2) and
UTF-32 entries say 8 (actual: 4).

The only consumer of that field is the length check on the cross-endian slow
path, so unpickling an odd-length 'w' array from a machine of the opposite
endianness fails:

import array
recon = array._array_reconstructor  # what pickle calls
data = 'abc'.encode('utf-32-be')    # 3 code points, big-endian
recon(array.array, 'w', 21, data)   # 21 = UTF32_BE, on a little-endian machine
# ValueError: string length not a multiple of item size

Even lengths decode correctly, and same-endian pickles use the fast path that
skips the check, which is why this stayed hidden. On 3.13/3.14 the 'u'
typecode maps to the (equally wrong) UTF-16 entries on Windows.

Linked PRs
  • gh-155851

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in Modules/arraymodule.c, especially the mformat_descriptors table and the _array_reconstructor cross-endian path. Verify that odd-length UTF-16 and UTF-32 'w' or 'u' data can be unpickled across endianness without the item-size ValueError; linked PR gh-155851 indicates the work is already underway.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.