cloudflare / cloudflare/sqlalchemy-clickhouse

pandas.DataFrame.to_sql using connector doesn't write the last row

Open
#10 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
325
Forks
101
PR merge metrics
No merged PRs in 30d

Description

There is a strange bug - when I try to use this connector to insert rows of pandas.DataFrame to ClickHouse.

```
import sqlalchemy as sa
import pandas

CLICKHOUSE_DSN = '***SECRET***'
ch_db = sa.create_engine(CLICKHOUSE_DSN)

pdf = pandas.DataFrame.from_records([
{'year': 1994, 'first_name': 'Vova'},
{'year': 1995, 'first_name': 'Anja'},
{'year': 1996, 'first_name': 'Vasja'},
{'year': 1997, 'first_name': 'Petja'},
])

pdf.to_sql('test_humans', ch_db, if_exists='append', index=False)
```

When I run it, I can see first 3 rows of pdf in table, but not the last one :-)
When I use postgres connector, or, this alternative clickhouse connector https://github.com/xzkostyan/clickhouse-sqlalchemy all works fine.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the example with pandas.DataFrame.to_sql, a SQLAlchemy engine, and the ClickHouse connector, then compare the inserted rows with the four source records. Trace the connector's insert path and verify why the final row is omitted; done means all four rows appear, as they do with the PostgreSQL and alternative ClickHouse connectors.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python, sqlalchemy
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.