blaze / blaze/odo

convert sas7bdat to hdfstore in chunks

Open
#541 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

Hi,

From the docs I didn't understood if odo is supposed to handle conversion of files that do not fit in memory automatically. I'm trying a sas7bdat -> hdfstore conversion of a really large file like this:

> odo('file.sas7bdat', 'hdfstore://file.h5::/data')

and memory usage is growing (apparently) indefinitely. I say apparently since I interrupt the process before the OS starts swapping.

I tried to use the "chunks" function, but I can't get the syntax right.

This, for example, doesn't work:

> odo(chunks(pandas.DataFrame)('file.sas7bdat'),'hdfstore://file.h5::/data')

Thanks!
Marco

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.