apache / apache/arrow

Add more pyarrow.type_for_alias(es) to support binary, decimals, etc.

Open
#43,858 3 comments 1 reaction 0 assignees View on GitHub
Component: C++ Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Can we add more aliases to:
https://github.com/apache/arrow/blob/main/python/pyarrow/types.pxi

I added some additional ones on my own, but it would be nice if we can make these standard?

I did this to support adding schema definitions into JSON and YAML config files. I needed string representations for these arrow types.

** Updated issue to reflect "decimal()" and "timestamp[precision, tz=]" aliases for precision and timezone **

```
"binary" to pa.binary()
"fixed_sized_binary[?]" to pa.binary(?)
"decimal128[?]" or "decimal128[?,?]" to pa.decimal128(?) or pa.decimal128(?,?)
"decimal256[?]" or "decimal256[?,?]" to pa.decimal256(?) or pa.decimal256(?,?)
"decimal[?]" or "decimal[?,?]" to pa.decimal256() if precision is > 38
"decimal[?]" or "decimal[?,?]" to pa.decimal128() if precision is <= 38
"timestamp" to pa.timestamp('us')
"date" to pa.date32()
"int" to pa.int32()
```

```
if obj.startswith("binary") or obj.startswith("fixed_size_binary"):
res = re.search(r"\[.*?\]", obj)
if res:
length = int(res.group()[1:-1])
obj = pa.binary(length)
else:
obj = pa.binary()
elif obj.startswith("decimal"):
res = re.search(r"\[.*?\]|\(.*?\)", obj)
scale_precision = []
if res:
scale_precision = [int(i) for i in res.group()[1:-1].split(",")]

if obj.startswith("decimal256"):
dec_func = pa.decimal256
elif obj.startswith("decimal128"):
dec_func = pa.decimal128
elif obj.startswith("decimal"):
if scale_precision[0] > 38:
dec_func = pa.decimal256
else:
dec_func = pa.decimal128

if len(scale_precision) == 1:
obj = dec_func(scale_precision[0])
elif len(scale_precision) == 2:
obj = dec_func(scale_precision[0], scale_precision[1])
elif obj.startswith("timestamp"):
# https://github.com/apache/arrow/blob/ab95a4d25142ff5723117c9d3a1c6453a6640cf6/python/pyarrow/src/arrow/python/datetime.h#L185
pattern = re.compile(
r"^timestamp\[(?Ps|ms|us|ns)(?:, ?tz=(?P[\w/\+-]+|(?:\+|-)\d{2}:\d{2}))?\]$"
)
res = re.match(pattern, obj)
obj = pa.timestamp(**res.groupdict()) if res else pa.timestamp("us")
else:
if obj == "date":
# default date to date32 which is an alias for date32[day]
obj = "date32"
elif obj == "int":
# default int to int32
obj = "int32"
obj = pa.type_for_alias(obj)
return obj

```

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.