apache / apache/arrow

[C++] Handle only relevant slices of child arrays when hashing scalars from ListArrays

Open
#35,830 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Issue is explained in the Python code below:

```python
import pyarrow as pa

a = pa.array([
[{'a': 5}, {'a': 6}],
[{'a': 7}, None]
])
b = pa.array([
[{'a': 7}, None]
])

# a[1] and b[0] are represented as 2-element slices of a child array containing struct values
# they start on different offsets, but obviously compare as equal
assert a[1] == b[0]

# logically equal values should hash to the same value, so when hashing the hashing
# of the child array should start at the offset and not from 0 as it's done by default.
hash1 = hash(a[1])
hash2 = hash(b[0])
assert hash1 == hash2
```

#35814 fixes the bug for lists of structs, but the same bug might exist for other nested types:

- [x] struct
- [ ] sparse union
- [ ] dense union
- [ ] run-end encoded
- [ ] more?

### Component(s)

C++

Contributor guide

Open the contributing guide

Research direction

Start with the Python reproducer and compare the behavior with the lists-of-structs fix referenced in #35814. Trace the C++ scalar hashing path for nested arrays, then check sparse union, dense union, and run-end encoded values. Done means logically equal slices produce identical hashes across the remaining relevant nested types.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.