ClickHouse / ClickHouse/clickhouse-connect
Improve Insert Performance
- Dominant language
- Python
- Stars
- 521
- Forks
- 159
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 43
Description
Insert data transformation is currently all in Python. This should be moved to C/Cython where expensive. It should also be possible to optimize Numpy/Pandas inserts by reading the Numpy buffer directly for int/float datatypes.
```[tasklist]
### Tasks
- [x] https://github.com/ClickHouse/clickhouse-connect/issues/201
- [ ] Direct copy of numpy arrays to output buffer
- [ ] Buffered bytearray destination for all writes to reduce allocations
- [ ] Datetime optimizations in C
- [ ] Decimal optimizations in C
- [x] Dynamically calculate block size
```
Contributor guide
Research direction
Start by reviewing the unchecked tasks in issue #106, especially direct NumPy array copying, buffered bytearray output, and C datetime and Decimal optimizations. Trace the Python insert transformation path and identify the existing work referenced by issue #201; done means the remaining listed tasks are implemented and the insert path no longer performs the targeted expensive transformations in Python.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, numpy, pandas, python
- Domain
- database, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100