ClickHouse / ClickHouse/clickhouse-connect

Improve Insert Performance

Open
#106 4 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
521
Forks
159
Avg merge
1d 8h
Merged PRs (30d)
43

Description

Insert data transformation is currently all in Python. This should be moved to C/Cython where expensive. It should also be possible to optimize Numpy/Pandas inserts by reading the Numpy buffer directly for int/float datatypes.
```[tasklist]
### Tasks
- [x] https://github.com/ClickHouse/clickhouse-connect/issues/201
- [ ] Direct copy of numpy arrays to output buffer
- [ ] Buffered bytearray destination for all writes to reduce allocations
- [ ] Datetime optimizations in C
- [ ] Decimal optimizations in C
- [x] Dynamically calculate block size
```

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the unchecked tasks in issue #106, especially direct NumPy array copying, buffered bytearray output, and C datetime and Decimal optimizations. Trace the Python insert transformation path and identify the existing work referenced by issue #201; done means the remaining listed tasks are implemented and the insert path no longer performs the targeted expensive transformations in Python.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, numpy, pandas, python
Domain
database, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.