convert sas7bdat to hdfstore in chunks
Open
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
From the docs I didn't understood if odo is supposed to handle conversion of files that do not fit in memory automatically. I'm trying a sas7bdat -> hdfstore conversion of a really large file like this:
> odo('file.sas7bdat', 'hdfstore://file.h5::/data')
and memory usage is growing (apparently) indefinitely. I say apparently since I interrupt the process before the OS starts swapping.
I tried to use the "chunks" function, but I can't get the syntax right.
This, for example, doesn't work:
> odo(chunks(pandas.DataFrame)('file.sas7bdat'),'hdfstore://file.h5::/data')
Thanks!
Marco
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.