apache / apache/arrow

[Python] Pyarrow DictionaryArray.dictionary_decode mangling strings

Open
#34,583 4 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I have an Numpy array of strings containing UK postcodes (2,603,678 entries) and another Numpy array of indices (1,785,499,246 entries) to them.

In the output file I want to replace the indices with the strings, so I created a DictionaryArray from them as follows:

````
postcode_dict = pa.DictionaryArray.from_arrays(pcds_id, pcds)
````

Where pcds_id contains the indices and pcds contains the postcode strings.

A UK postcode format is A[A]N[N|A] NAA so varies between 6 and 8 characters in length.

A = alpha, N = numeric, | = or, [] = optional.

The dictionary creation works file:

````

-- dictionary:
[
"AB1 0AA",
"AB1 0AB",
"AB1 0AD",
"AB1 0AE",
"AB1 0AF",
"AB1 0AG",
"AB1 0AJ",
"AB1 0AL",
"AB1 0AN",
"AB1 0AP",
...
"ZE3 9JP",
"ZE3 9JR",
"ZE3 9JS",
"ZE3 9JT",
"ZE3 9JU",
"ZE3 9JW",
"ZE3 9JX",
"ZE3 9JY",
"ZE3 9JZ",
"ZE3 9XP"
]
-- indices:
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
...
2603676,
2603676,
2603677,
2603677,
2603677,
2603677,
2603677,
2603677,
2603677,
2603677
]
````

However applying dictionary_decode to this array results in the strings becoming mangled:

````

[
"AB1 0AA",
"AB1 0AA",
"AB1 0AA",
"AB1 0AA",
"AB1 0AA",
"AB1 0AA",
"AB1 0AA",
"AB1 0AA",
"AB1 0AA",
"AB1 0AA",
...
" 1UGAL1",
" 1UGAL1",
" 1UGAL1",
" 1UGAL1",
" 1UGAL1",
" 1UGAL1",
" 1UGAL1",
" 1UGAL1",
" 1UGAL1",
" 1UGAL1"
]
````

However, if I convert the array to pandas it formats correctly:

````
postcode_dict.to_pandas()

0 AB1 0AA
1 AB1 0AA
2 AB1 0AA
3 AB1 0AA
4 AB1 0AA
...
1785499241 ZE3 9XP
1785499242 ZE3 9XP
1785499243 ZE3 9XP
1785499244 ZE3 9XP
1785499245 ZE3 9XP
Length: 1785499246, dtype: category
Categories (2603678, object): ['AB1 0AA', 'AB1 0AB', 'AB1 0AD', 'AB1 0AE', ..., 'ZE3 9JX', 'ZE3 9JY', 'ZE3 9JZ', 'ZE3 9XP']
````

Source arrays in NPZ format inside a zip file:

[postcode_dict.zip](https://github.com/apache/arrow/files/10988887/postcode_dict.zip)

DictionaryArray in parquet inside a zip file on Google Drive (too large for Github):

[postcode_dict_parquet.zip](https://drive.google.com/file/d/1X4SiljQ2bOW8VUM5yjk32VDrI22MGEvR/view?usp=sharing)

I'm using a RAPIDS Docker file from NVIDIA NGC (as there are GPU dependencies, cuGraph, in the workflow) and the Pyarrow version is '10.0.1'

### Component(s)

Python

### Data Source Attribution & License

Data is a spatial weights matrix (crosstab of graph distances) built from UK Office for National Statistics [ONS Postcode Directory February 2023](https://geoportal.statistics.gov.uk/datasets/ons-postcode-directory-february-2023-version-2/about)

- Contains OS data © Crown copyright and database right 2023
- Contains Royal Mail data © Royal Mail copyright and database right 2023
- Source: Office for National Statistics licensed under the Open Government Licence v.3.0

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the attached NPZ input and PyArrow 10.0.1, focusing on DictionaryArray.from_arrays followed by dictionary_decode. Compare the decoded values with DictionaryArray.to_pandas(); done means dictionary_decode preserves the original postcode strings for the supplied indices.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.