apache / apache/arrow

[Python] Change StructArray.field(..) to return "flattened" field?

Open
#14,970 0 comments 0 reactions 0 assignees View on GitHub
Component: Python
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Related to https://github.com/apache/arrow/issues/14946 on the C++ side, and this recently came up in https://github.com/apache/arrow/pull/14781#issuecomment-1339114243.

A StructArray has child arrays that make up its "fields", but in addition it can also have a top-level validity bitmap. So when accessing a field of a StructArray that has such top-level nulls, you can retrieve the "raw" child array or you can get the "logical" field array that combines the child array with the top-level bitmap.

To illustrate:

```
In [1]: arr = pa.StructArray.from_arrays([pa.array([5, 3, 4, 2, 1]), pa.array([1, 2, 3, 4, 5])], names=['a', 'b'], mask=pa.array([False, True, False, False, False]))

In [2]: arr.to_pandas()
Out[2]:
0 {'a': 5, 'b': 1}
1 None
2 {'a': 4, 'b': 3}
3 {'a': 2, 'b': 4}
4 {'a': 1, 'b': 5}
dtype: object

In [3]: arr.field('a')
Out[3]:

[
5,
3,
4,
2,
1
]

In [4]: arr.flatten()[0]
Out[4]:

[
5,
null,
4,
2,
1
]
```

Currently, the `field()` method on a StructArray gives you the raw child array, and there is a `flatten()` method that returns those "logical" field arrays for all the fields as a list of arrays.
We should have a method with which you can get the field array for a single field instead of having to use `flatten()`, and in https://github.com/apache/arrow/pull/14781, @amol- added a `_flattened_field` (private for now, but we needed it to get the correct values to sort by):

```
In [5]: arr._flattened_field('a')
Out[5]:

[
5,
null,
4,
2,
1
]
```

We could just make that a public method instead, however, some questions/concerns about this:

- I personally don't like the "flattened" term. I know we already use this in C++ as well (this basically just exposes the C++ `StructArray::GetFlattenedField`), but I don't find it very clear that it means this distinction.
- We could also change `field()` instead? I personally think this is what people typically will want when they currently call `field` (like @amol- was doing in the sort PR, to get the values of a certain field of the struct). The value in the raw child that is being masked by the top-level bitmap is kind of an implementation detail, and IMO a user should not necessarily get that so easily.
- If we would change `field()` to default to the "flattened" field, we need an alternative to access the raw child. We could add a keyword for this? (but what name?) Or a separate method like `child()`?

Contributor guide

Open the contributing guide

Research direction

Start with the Python StructArray.field(), flatten(), and private _flattened_field() entry points described here, comparing raw-child and logical-field behavior when top-level validity is present. Resolve whether to expose a new public method, change field(), or add a raw-child alternative; done means the selected API and masking behavior are defined and implemented.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.