alteryx / alteryx/woodwork

Spike: Investigate using native PyArrow backed data types

オープン
#1,520 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
spike
主要言語
Python
スター
155
フォーク
24
PR マージ指標
30日以内にマージされた PR はありません

説明

- With the upcoming pandas 1.5.0 release, pandas will support most data types with pyarrow backed Arrays and Data Types (https://pandas.pydata.org/docs/dev/whatsnew/v1.5.0.html#native-pyarrow-backed-extensionarray)
- TLDR of using Apache Arrow:
- faster computations, less memory usage --> Woodwork DataFrames are faster & take up less memory

## Initial Investigation
### Install
```shell
pip install pandas==1.5.0rc0
```
```shell
pip install pyarrow --upgrade
```

### Mapping
```python
import pandas as pd
import pyarrow as pa
from pandas.core.arrays.arrow.dtype import ArrowDtype

PANDAS_DTYPE_TO_ARROWDTYPE = {
"np.int64": pa.int64(),
"int64": pa.int64(),
"Int64": pa.int64(),
"np.float64": pa.float64(),
"float64": pa.float64(),
"Float64Dtype": pa.float64(),
"np.object": pa.string(),
"object": pa.string(),
"string": pa.string(),
"StringDtype": pa.string(),
"datetime64[s]": pa.timestamp(unit="s", tz=None),
"datetime64[s, US/Eastern]": pa.timestamp(unit="s", tz='US/Eastern'),
"datetime64[s, US/Central]": pa.timestamp(unit="s", tz='US/Central'),
"datetime64[ms]": pa.timestamp(unit="ms", tz=None),
"datetime64[ms, US/Eastern]": pa.timestamp(unit="ms", tz='US/Eastern'),
"datetime64[ms, US/Central]": pa.timestamp(unit="ms", tz='US/Central'),
"datetime64[us]": pa.timestamp(unit="us", tz=None),
"datetime64[us, US/Eastern]": pa.timestamp(unit="us", tz='US/Eastern'),
"datetime64[us, US/Central]": pa.timestamp(unit="us", tz='US/Central'),
"datetime64[ns]": pa.timestamp(unit="ns", tz=None),
"datetime64[ns, US/Eastern]": pa.timestamp(unit="ns", tz='US/Eastern'),
"datetime64[ns, US/Central]": pa.timestamp(unit="ns", tz='US/Central'),
"np.bool_": pa.bool_(),
"boolean": pa.bool_(),
"BooleanDtype": pa.bool_()
}
for k, v in PANDAS_DTYPE_TO_ARROWDTYPE.items():
PANDAS_DTYPE_TO_ARROWDTYPE[k] = ArrowDtype(v)
```

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。