microsoft / microsoft/mssql-python
Optimization: Can we use chunking for String ( CHAR,VARCAR,NVARCHAR) data type to remove extra memory allocation
@subrata-ms is already working on this.
Since Feb 26, 2026.
- Dominant language
- Python
- Stars
- 472
- Forks
- 60
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 36
Description
Describe the bug
Currently we allocate fixed size memory ( coulumnsize+1) to handle string data type. This was causing problem for CP1252 character set. We have fix this issue under reported SQLAlchemy bug ( https://github.com/microsoft/mssql-python/issues/435?reload=1?reload=1 ).
However, we do see an opportunity improve the memory allocation for the string data type. One of the consideration/approach could be chunking.
This will help to reduce unnecessary memory allocation considerably during multithreaded execution and for large dataset.
Exception message:
Stack trace:
To reproduce
This is considered as code optimization. Below is the related bug -
https://github.com/microsoft/mssql-python/issues/435?reload=1?reload=1
Expected behavior
All string data type should handle CP1252 character set.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.