Support for pandas `_query_iterator`
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
It would be nice if this worked:
``` python
gen = pd.read_sql_query(complex_sql, engine, chunksize=10)
odo.into('data.hdf5::/test', gen)
```
Right now I don't think `odo` is aware that `gen`, a `_query_iterator` is just an iterator of DataFrames. You get back that `_query_iterator` from `pd.read_sql_table` or `pd.read_sql_query` when you specify a `chunksize`.
I used a workaround Matthew suggested on the mailing list
``` python
def f():
gen = pd.read_sql_query(complex_sql, engine, chunksize=10)
for df in gen:
yield df
odo.into('data.hdf5::/test', odo.chunks(pd.DataFrame)(f))
```
For testing, this doesn't even need to be a complex query, you can just use `pd.read_sql_table` with a chunksize.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.