apache / apache/arrow

[C++][Compute] unique, value_counts and dictionary_encode have no kernel for float16 (halffloat)

Open
#50,512 0 comments 0 reactions 1 assignee Claimed by @fornwall View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

The hash-based vector functions do not support `halffloat` input, while supporting all other numeric types:

```python
>>> import pyarrow as pa
>>> pa.array([1.5, 2.5], type=pa.float16()).dictionary_encode()
ArrowNotImplementedError: Function 'dictionary_encode' has no kernel matching input types (halffloat)
```

`unique` and `value_counts` fail the same way. Reproduced on current `main`; platform-independent (missing kernel registration, not a platform issue).

The cause is that `AddHashKernels()` in `cpp/src/arrow/compute/kernels/vector_hash.cc` registers kernels by iterating `PrimitiveTypes()`, which excludes float16 (`FloatingPointTypes()` is only `{float32(), float64()}`), and `GetHashInit()` has no `Type::HALF_FLOAT` case.

This was previously part of the broader GH-43017 ("Make the set of casts and hash kernels involving float16 consistent with other floating types", closed as stale); its PR #43018 stalled on the cast/sort parts, not the hash-kernel part. This issue proposes fixing only the hash kernels, registering float16 explicitly the same way recent merged float16 additions did (#46446 for if_else/replace, #46286 for run_end_encode), i.e. without widening `FloatingPointTypes()`/`NumericTypes()`/`PrimitiveTypes()`, whose other consumers are templated on the `uint16_t` storage type and would miscompute for half floats.

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.