blaze / blaze/odo

Support for pandas `_query_iterator`

Open
#145 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

It would be nice if this worked:

``` python
gen = pd.read_sql_query(complex_sql, engine, chunksize=10)
odo.into('data.hdf5::/test', gen)
```

Right now I don't think `odo` is aware that `gen`, a `_query_iterator` is just an iterator of DataFrames. You get back that `_query_iterator` from `pd.read_sql_table` or `pd.read_sql_query` when you specify a `chunksize`.

I used a workaround Matthew suggested on the mailing list

``` python
def f():
gen = pd.read_sql_query(complex_sql, engine, chunksize=10)
for df in gen:
yield df

odo.into('data.hdf5::/test', odo.chunks(pd.DataFrame)(f))
```

For testing, this doesn't even need to be a complex query, you can just use `pd.read_sql_table` with a chunksize.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.