Spike: Investigate using native PyArrow backed data types
- 主要言語
- Python
- スター
- 155
- フォーク
- 24
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
- With the upcoming pandas 1.5.0 release, pandas will support most data types with pyarrow backed Arrays and Data Types (https://pandas.pydata.org/docs/dev/whatsnew/v1.5.0.html#native-pyarrow-backed-extensionarray)
- TLDR of using Apache Arrow:
- faster computations, less memory usage --> Woodwork DataFrames are faster & take up less memory
## Initial Investigation
### Install
```shell
pip install pandas==1.5.0rc0
```
```shell
pip install pyarrow --upgrade
```
### Mapping
```python
import pandas as pd
import pyarrow as pa
from pandas.core.arrays.arrow.dtype import ArrowDtype
PANDAS_DTYPE_TO_ARROWDTYPE = {
"np.int64": pa.int64(),
"int64": pa.int64(),
"Int64": pa.int64(),
"np.float64": pa.float64(),
"float64": pa.float64(),
"Float64Dtype": pa.float64(),
"np.object": pa.string(),
"object": pa.string(),
"string": pa.string(),
"StringDtype": pa.string(),
"datetime64[s]": pa.timestamp(unit="s", tz=None),
"datetime64[s, US/Eastern]": pa.timestamp(unit="s", tz='US/Eastern'),
"datetime64[s, US/Central]": pa.timestamp(unit="s", tz='US/Central'),
"datetime64[ms]": pa.timestamp(unit="ms", tz=None),
"datetime64[ms, US/Eastern]": pa.timestamp(unit="ms", tz='US/Eastern'),
"datetime64[ms, US/Central]": pa.timestamp(unit="ms", tz='US/Central'),
"datetime64[us]": pa.timestamp(unit="us", tz=None),
"datetime64[us, US/Eastern]": pa.timestamp(unit="us", tz='US/Eastern'),
"datetime64[us, US/Central]": pa.timestamp(unit="us", tz='US/Central'),
"datetime64[ns]": pa.timestamp(unit="ns", tz=None),
"datetime64[ns, US/Eastern]": pa.timestamp(unit="ns", tz='US/Eastern'),
"datetime64[ns, US/Central]": pa.timestamp(unit="ns", tz='US/Central'),
"np.bool_": pa.bool_(),
"boolean": pa.bool_(),
"BooleanDtype": pa.bool_()
}
for k, v in PANDAS_DTYPE_TO_ARROWDTYPE.items():
PANDAS_DTYPE_TO_ARROWDTYPE[k] = ArrowDtype(v)
```
コントリビューションガイド
評価
この issue はまだ評価されていません。