Add more pyarrow.type_for_alias(es) to support binary, decimals, etc.
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Can we add more aliases to:
https://github.com/apache/arrow/blob/main/python/pyarrow/types.pxi
I added some additional ones on my own, but it would be nice if we can make these standard?
I did this to support adding schema definitions into JSON and YAML config files. I needed string representations for these arrow types.
** Updated issue to reflect "decimal()" and "timestamp[precision, tz=]" aliases for precision and timezone **
```
"binary" to pa.binary()
"fixed_sized_binary[?]" to pa.binary(?)
"decimal128[?]" or "decimal128[?,?]" to pa.decimal128(?) or pa.decimal128(?,?)
"decimal256[?]" or "decimal256[?,?]" to pa.decimal256(?) or pa.decimal256(?,?)
"decimal[?]" or "decimal[?,?]" to pa.decimal256() if precision is > 38
"decimal[?]" or "decimal[?,?]" to pa.decimal128() if precision is <= 38
"timestamp" to pa.timestamp('us')
"date" to pa.date32()
"int" to pa.int32()
```
```
if obj.startswith("binary") or obj.startswith("fixed_size_binary"):
res = re.search(r"\[.*?\]", obj)
if res:
length = int(res.group()[1:-1])
obj = pa.binary(length)
else:
obj = pa.binary()
elif obj.startswith("decimal"):
res = re.search(r"\[.*?\]|\(.*?\)", obj)
scale_precision = []
if res:
scale_precision = [int(i) for i in res.group()[1:-1].split(",")]
if obj.startswith("decimal256"):
dec_func = pa.decimal256
elif obj.startswith("decimal128"):
dec_func = pa.decimal128
elif obj.startswith("decimal"):
if scale_precision[0] > 38:
dec_func = pa.decimal256
else:
dec_func = pa.decimal128
if len(scale_precision) == 1:
obj = dec_func(scale_precision[0])
elif len(scale_precision) == 2:
obj = dec_func(scale_precision[0], scale_precision[1])
elif obj.startswith("timestamp"):
# https://github.com/apache/arrow/blob/ab95a4d25142ff5723117c9d3a1c6453a6640cf6/python/pyarrow/src/arrow/python/datetime.h#L185
pattern = re.compile(
r"^timestamp\[(?Ps|ms|us|ns)(?:, ?tz=(?P[\w/\+-]+|(?:\+|-)\d{2}:\d{2}))?\]$"
)
res = re.match(pattern, obj)
obj = pa.timestamp(**res.groupdict()) if res else pa.timestamp("us")
else:
if obj == "date":
# default date to date32 which is an alias for date32[day]
obj = "date32"
elif obj == "int":
# default int to int32
obj = "int32"
obj = pa.type_for_alias(obj)
return obj
```
### Component(s)
C++, Python
Contributor guide
Assessment
This issue has not been assessed yet.