array: unpickling an odd-length 'w' array across endianness raises ValueError (wrong UTF-16/UTF-32 item size)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
Bug report
array.array pickles carry a machine-format code so the receiver can decode
data from a different platform. The decoder's format table
(mformat_descriptors in Modules/arraymodule.c) stores a wrong item size
for the Unicode formats: UTF-16 entries say 4 bytes per item (actual: 2) and
UTF-32 entries say 8 (actual: 4).
The only consumer of that field is the length check on the cross-endian slow
path, so unpickling an odd-length 'w' array from a machine of the opposite
endianness fails:
import array
recon = array._array_reconstructor # what pickle calls
data = 'abc'.encode('utf-32-be') # 3 code points, big-endian
recon(array.array, 'w', 21, data) # 21 = UTF32_BE, on a little-endian machine
# ValueError: string length not a multiple of item size
Even lengths decode correctly, and same-endian pickles use the fast path that
skips the check, which is why this stayed hidden. On 3.13/3.14 the 'u'
typecode maps to the (equally wrong) UTF-16 entries on Windows.
Linked PRs
- gh-155851
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in Modules/arraymodule.c, especially the mformat_descriptors table and the _array_reconstructor cross-endian path. Verify that odd-length UTF-16 and UTF-32 'w' or 'u' data can be unpickled across endianness without the item-size ValueError; linked PR gh-155851 indicates the work is already underway.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100