[Question] save to a .jay file from a python iterator (with dict. as elements)
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 164
- Avg merge
- 7h 31m
- Merged PRs (30d)
- 1
Description
Dear datatable developers:
I have a large (not super large though) hdf5 file. I want to convert it to .jay file, but I prefer not to load the whole data in hdf5 into memory, create a Frame than save to .jay.
I am wondering if it is possible to save to .jay by specifying an iterator of dictionaries.
From [https://datatable.readthedocs.io/en/latest/api/frame/__init__.html](https://datatable.readthedocs.io/en/latest/api/frame/__init__.html), I see
```
If the source is a list of dict objects, then each element in this list is interpreted as a single row.
The keys in each dictionary are column names, and the values contain contents of each individual cell.
```
But I don't want to make the full list of rows from hdf5, since I don't want to load them into memory.
** Update
Finally, I tried to load the data inside hdf5 to a numpy array in memory, created a `Frame`. Then tried to save to .jay.
The numpy array takes 10GB memory, and during the call `to_jay()`, the system memory usage went up to 16GB and crashed ...
If this is not possible, another way might be to specify a list of iterators, each of them iterates a single column.
It would be great if `datatable` can do such thing. Thank you.
Contributor guide
Assessment
This issue has not been assessed yet.