h2oai / h2oai/datatable

extremely slow when setting multiindex key

Open
#2,091 2 comments 0 reactions 0 assignees View on GitHub
performance sort
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

When I was trying to set multiindex key to a dataframe with more than 10M rows, most of the time I can observe CPU overshotting and the process would take more than 5mins (which supposed to be less than 10s).

How to reproduce: `df` is `pd.DataFrame` object with 10M rows.
```
frame = dt.Frame(df)
frame.key = ('Date', 'Symbol')
```

When it works normally, it would take ~5secs. When leakage observed, more than 5mins.

Environment:

```
commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-65-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
pandas: 0.24.2
pytest: 5.0.1
pip: 19.1.1
setuptools: 41.0.1
Cython: None
numpy: 1.16.4
scipy: 1.3.1
pyarrow: 0.12.1
xarray: None
IPython: 7.6.1
sphinx: None
patsy: None
dateutil: 2.8.0
pytz: 2019.1
blosc: None
bottleneck: None
tables: 3.5.2
numexpr: 2.7.0
feather: 0.4.0
matplotlib: 3.1.0
openpyxl: 2.6.2
xlrd: 1.2.0
xlwt: None
xlsxwriter: 1.1.8
lxml.etree: 4.4.1
bs4: 4.8.0
html5lib: None
sqlalchemy: 1.3.8
pymysql: 0.9.3
psycopg2: None
jinja2: 2.10.1
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.