h2oai / h2oai/datatable

[Question] save to a .jay file from a python iterator (with dict. as elements)

Open
#2,845 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

Dear datatable developers:

I have a large (not super large though) hdf5 file. I want to convert it to .jay file, but I prefer not to load the whole data in hdf5 into memory, create a Frame than save to .jay.

I am wondering if it is possible to save to .jay by specifying an iterator of dictionaries.

From [https://datatable.readthedocs.io/en/latest/api/frame/__init__.html](https://datatable.readthedocs.io/en/latest/api/frame/__init__.html), I see

```
If the source is a list of dict objects, then each element in this list is interpreted as a single row.
The keys in each dictionary are column names, and the values contain contents of each individual cell.
```
But I don't want to make the full list of rows from hdf5, since I don't want to load them into memory.

** Update
Finally, I tried to load the data inside hdf5 to a numpy array in memory, created a `Frame`. Then tried to save to .jay.
The numpy array takes 10GB memory, and during the call `to_jay()`, the system memory usage went up to 16GB and crashed ...

If this is not possible, another way might be to specify a list of iterators, each of them iterates a single column.

It would be great if `datatable` can do such thing. Thank you.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.