cloudflare / cloudflare/sqlalchemy-clickhouse
pandas.DataFrame.to_sql using connector doesn't write the last row
- Dominant language
- Python
- Stars
- 325
- Forks
- 101
- PR merge metrics
- No merged PRs in 30d
Description
There is a strange bug - when I try to use this connector to insert rows of pandas.DataFrame to ClickHouse.
```
import sqlalchemy as sa
import pandas
CLICKHOUSE_DSN = '***SECRET***'
ch_db = sa.create_engine(CLICKHOUSE_DSN)
pdf = pandas.DataFrame.from_records([
{'year': 1994, 'first_name': 'Vova'},
{'year': 1995, 'first_name': 'Anja'},
{'year': 1996, 'first_name': 'Vasja'},
{'year': 1997, 'first_name': 'Petja'},
])
pdf.to_sql('test_humans', ch_db, if_exists='append', index=False)
```
When I run it, I can see first 3 rows of pdf in table, but not the last one :-)
When I use postgres connector, or, this alternative clickhouse connector https://github.com/xzkostyan/clickhouse-sqlalchemy all works fine.
Contributor guide
Research direction
Start by reproducing the example with pandas.DataFrame.to_sql, a SQLAlchemy engine, and the ClickHouse connector, then compare the inserted rows with the four source records. Trace the connector's insert path and verify why the final row is omitted; done means all four rows appear, as they do with the PostgreSQL and alternative ClickHouse connectors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python, sqlalchemy
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100