blaze / blaze/odo

HDFStore discover hangs on large store

Open
#582 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

Hi,

I'm trying to use odo to pull data out from an HDFStore file (eventually destined to a sqlite database, though that's not relevant now. Here's the code I'm running:

```python
from odo import *
d = resource("hdfstore://data.h5")
shape = discover(d)
```

The process pegs at 100% cpu but only 1.3GB virtual memory and 100mb resident. Here's the backtrace I get when I hit ctrl-c:

Backtrace on KeyboardInterrupt

In [5]: shape = discover(d)
cc^[[B^C---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)
in ()
----> 1 shape = discover(d)

/nfs/projects_nobackup/c/cidgrowlab/Mali/intl-atlas-api/env/lib/python3.4/site-packages/multipledispatch/dispatcher.py in __call__(self, *args, **kwargs)
162 self._cache[types] = func
163 try:
--> 164 return func(*args, **kwargs)
165
166 except MDNotImplementedError:

/nfs/projects_nobackup/c/cidgrowlab/Mali/intl-atlas-api/env/lib/python3.4/site-packages/odo/backends/hdfstore.py in discover_hdfstore(f)
21 while '/' in key2:
22 group, key2 = key2.split('/', 1)
---> 23 if group not in d2:
24 d2[group] = dict()
25 d2 = d2[group]

/nfs/projects_nobackup/c/cidgrowlab/Mali/intl-atlas-api/env/lib/python3.4/site-packages/pandas/io/pytables.py in __getitem__(self, c)
2997 def __getitem__(self, c):
2998 """ return the axis for c """
-> 2999 for a in self.axes:
3000 if c == a.name:
3001 return a

/nfs/projects_nobackup/c/cidgrowlab/Mali/intl-atlas-api/env/lib/python3.4/site-packages/pandas/io/pytables.py in axes(self)
3080 @property
3081 def axes(self):
-> 3082 return itertools.chain(self.index_axes, self.values_axes)
3083
3084 @property

KeyboardInterrupt:

My uneducated guess is that it's doing a chunked scan through the file, but is this strictly necessary? I thought that it should be possible to get the shape metadata from the HDF file itself. Here's some info on the file (sizes and shapes), it's large but not huge:

```
(env) [makmanalp@cod6-ksg-1 intl-atlas-api]$ pttree -L2 --print-shape data.h5

------------------------------------------------------------

/ (RootGroup)
+--country_partner_sitcproduct_year (Group)
| +--table (Table)
| | mem=3.7GiB, disk=664.1MiB [46.7%]
| | shape=(129718932,)
| `--meta (Group)
| ... 4 leaves, mem=20.7KiB, disk=4.0KiB [ 0.0%]
+--country_partner_hsproduct_year (Group)
| +--table (Table)
| | mem=3.1GiB, disk=547.6MiB [38.5%]
| | shape=(106940438,)
| `--meta (Group)
| ... 4 leaves, mem=27.3KiB, disk=4.0KiB [ 0.0%]
+--country_sitcproduct_year (Group)
| +--table (Table)
| | mem=275.0MiB, disk=118.8MiB [ 8.4%]
| | shape=(7793970,)
| `--meta (Group)
| ... 3 leaves, mem=16.9KiB, disk=3.1KiB [ 0.0%]
+--country_hsproduct_year (Group)
| +--table (Table)
| | mem=192.6MiB, disk=82.2MiB [ 5.8%]
| | shape=(5459306,)
| `--meta (Group)
| ... 3 leaves, mem=23.5KiB, disk=3.0KiB [ 0.0%]
+--country_partner_year (Group)
| +--table (Table)
| | mem=34.0MiB, disk=6.4MiB [ 0.4%]
| | shape=(1229844,)
| `--meta (Group)
| ... 3 leaves, mem=7.9KiB, disk=2.5KiB [ 0.0%]
+--classifications (Group)
| +--hs_product (Group)
| | ... 1 leaves, mem=7.6MiB, disk=458.7KiB [ 0.0%]
| +--sitc_product (Group)
| | ... 1 leaves, mem=756.7KiB, disk=57.4KiB [ 0.0%]
| `--location (Group)
| ... 1 leaves, mem=105.4KiB, disk=13.7KiB [ 0.0%]
+--sitcproduct_year (Group)
| `--table (Table)
| mem=1.9MiB, disk=518.6KiB [ 0.0%]
| shape=(42498,)
+--hsproduct_year (Group)
| `--table (Table)
| mem=1.2MiB, disk=323.5KiB [ 0.0%]
| shape=(26061,)
+--country_year (Group)
| `--table (Table)
| mem=297.8KiB, disk=91.7KiB [ 0.0%]
| shape=(5082,)
`--year (Group)
+--table (Table)
| mem=918.0B, disk=778.0B [ 0.0%]
| shape=(54,)
`--meta (Group)
... 1 leaves, mem=864.0B, disk=635.0B [ 0.0%]

------------------------------------------------------------
Total branch leaves: 30
Total branch size: 7.3GiB in memory, 1.4GiB on disk
Mean compression ratio: 0.19
HDF5 file size: 1.4GiB
------------------------------------------------------------
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at odo/backends/hdfstore.py, specifically discover_hdfstore, and reproduce the issue with resource("hdfstore://data.h5") followed by discover(d) on the supplied large store. Read the pandas HDFStore calls shown in the traceback to determine why discovery hangs. Done means discovery returns the store metadata and shape without an impractical wait or full data scan.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
Half a day
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.