apache / apache/beam

[Feature Request]: Some pyarrow numeric types are not supported in `arrow_type_compatibility`

Open
#23,816 0 comments 0 reactions 0 assignees View on GitHub
core new feature P2 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

### What would you like to happen?

Beam schemas do not support unsigned ints or float16s, so we currently refuse to map those in `arrow_type_compatibility`. If these types can't be added to core Beam schemas we might consider adding them as logical types.

### Issue Priority

Priority: 2

### Issue Component

Component: sdk-py-core

Contributor guide

Open the contributing guide

Research direction

Start by locating the `arrow_type_compatibility` entry point and the Beam schema handling it uses. Determine whether unsigned integers and float16 should be represented by core schemas or logical types, then identify the relevant compatibility tests. Done means the supported numeric types map consistently without being refused.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.