[Python] Consider splitting _lib module into several parts
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
When reading the logs of a wheel build on Windows I noticed these lines:
```
lib.cpp
C:\Python312\Lib\site-packages\numpy\_core\include\numpy\ndarraytypes.h(1250,38): warning C4200: nonstandard extension used: zero-sized array in struct/union [C:\arrow\python\build\temp.win-amd64-cpython-312\lib.vcxproj]
C:\Python312\Lib\site-packages\numpy\_core\include\numpy\ndarraytypes.h(1250,38): message : This member will be ignored by a defaulted constructor or copy/move assignment operator [C:\arrow\python\build\temp.win-amd64-cpython-312\lib.vcxproj]
C:\arrow\python\build\temp.win-amd64-cpython-312\lib.cpp(335145,17): warning C4244: '=': conversion from 'Py_ssize_t' to 'long', possible loss of data [C:\arrow\python\build\temp.win-amd64-cpython-312\lib.vcxproj]
C:\arrow\python\build\temp.win-amd64-cpython-312\lib.cpp(335645,17): warning C4244: '=': conversion from 'Py_ssize_t' to 'long', possible loss of data [C:\arrow\python\build\temp.win-amd64-cpython-312\lib.vcxproj]
C:\arrow\python\build\temp.win-amd64-cpython-312\lib.cpp(335855,17): warning C4244: '=': conversion from 'Py_ssize_t' to 'long', possible loss of data [C:\arrow\python\build\temp.win-amd64-cpython-312\lib.vcxproj]
```
Ignoring what the warnings say, what stands out is that the `lib.cpp` generated by Cython has *at least* 335000 lines. This is huge and can obviously lead to enormous compile times, especially if the RAM is not large enough for the C++ compiler to hold the entire intermediate representation(s) in memory.
We should definitely try to split the `_lib` into smaller parts, in order to alleviate this problem.
(it is also a Cython problem that so much C++ code is generated, but I'm not sure we can fix that).
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.