[Python] Failed to compile ARM64 PyArrow for Windows ARM
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
May I get your support on compiling ARM64 PyArrow.
I’m trying to compile it through the below steps on Windows on ARM device. When I run the last line command ‘python setup.py build_ext –inplace’ in step 5 to compile the PyArrow(Python extension), I get many error messages like below:
```
python_to_arrow.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) public: __cdecl arrow::LargeBinaryBuilder::LargeBinaryBuilder(class std::shared_ptr const &,class arrow::MemoryPool *)" (__imp_??0LargeBinaryBuilder@arrow@@QEAA@AEBV?$shared_ptr@VDataType@arrow@@@std@@PEAVMemoryPool@1@@Z) referenced in function "protected: virtual class arrow::Status __cdecl arrow::internal::PrimitiveConverter::Init(class arrow::MemoryPool *)" (?Init@?$PrimitiveConverter@VLargeBinaryType@arrow@@VPyConverter@
?A0xc3172271@py@2@@internal@arrow@@MEAA?AVStatus@3@PEAVMemoryPool@3@@Z) [C:\source\arrow\python\build\temp.win-arm64-cpython-312\arrow_python.vcxproj]
```
By comparing the ‘DUMPBIN’ result of ‘arrow.lib’ between x64 & arm64, the ARM64 lib missed the function with the parameter 'class std::shared_ptr const &,class arrow::MemoryPool *' in 'arrow.lib':
LargeBinaryBuilder is a template class. Do you have idea why the function missed while compiling it to ARM64? And how to fix it? Thanks in advance!
```
/// \class LargeBinaryBuilder
/// \brief Builder class for large variable-length binary data
class ARROW_EXPORT LargeBinaryBuilder : public BaseBinaryBuilder {
public:
using BaseBinaryBuilder::BaseBinaryBuilder;
/// \cond FALSE
using ArrayBuilder::Finish;
/// \endcond
Status Finish(std::shared_ptr* out) { return FinishTyped(out); }
std::shared_ptr type() const override { return large_binary(); }
};
```
Due to the reason mentioned in below link, I compiled the Cpp library with clang-cl for ARM64, but compiled the x64 version with MSVC cl.
https://arrow.apache.org/docs/developers/cpp/windows.html#building-on-windows-arm64-using-ninja-and-clang
The detailed steps:
```
1. Install ARM64 Python 3.12.6 and necessary Python extension.
2. Open "ARM64 Native Tools Command Prompt for VS 2022" command line and run below commands:
cd C:\source
git clone https://github.com/apache/arrow.git
3. Add below patch:
Patch 1:
diff --git a/cpp/src/arrow/CMakeLists.txt b/cpp/src/arrow/CMakeLists.txt
index c911f0f4e..ddd4dc0bb 100644
--- a/cpp/src/arrow/CMakeLists.txt
+++ b/cpp/src/arrow/CMakeLists.txt
@@ -955,7 +955,7 @@ if(CXX_LINKER_SUPPORTS_VERSION_SCRIPT)
endif()
if(ARROW_BUILD_STATIC AND ARROW_BUNDLED_STATIC_LIBS)
- set(ARROW_BUILD_BUNDLED_DEPENDENCIES TRUE)
+ set(ARROW_BUILD_BUNDLED_DEPENDENCIES FALSE)
else()
set(ARROW_BUILD_BUNDLED_DEPENDENCIES FALSE)
endif()
Patch 2:
diff --git a/python/setup.py b/python/setup.py
index 60b9a696d..6cde02919 100755
--- a/python/setup.py
+++ b/python/setup.py
@@ -165,7 +165,7 @@ class build_ext(_build_ext):
_build_ext.initialize_options(self)
self.cmake_generator = os.environ.get('PYARROW_CMAKE_GENERATOR')
if not self.cmake_generator and sys.platform == 'win32':
- self.cmake_generator = 'Visual Studio 15 2017 Win64'
+ self.cmake_generator = 'Visual Studio 17 2022'
self.extra_cmake_args = os.environ.get('PYARROW_CMAKE_OPTIONS', '')
self.build_type = os.environ.get('PYARROW_BUILD_TYPE',
'release').lower()
4. Run commands below to compile arrow Cpp code:
set ARROW_HOME=C:\source\arrow\Install
set CMAKE_PREFIX_PATH=C:\source\arrow\Install
mkdir arrow\cpp\build
pushd arrow\cpp\build
set CC=clang-cl
set CXX=clang-cl
cmake -G "Ninja" -DCMAKE_INSTALL_PREFIX=%ARROW_HOME% -DCMAKE_UNITY_BUILD=ON -DARROW_COMPUTE=ON -DARROW_CSV=ON -DARROW_DATASET=ON -DARROW_FILESYSTEM=ON -DARROW_HDFS=ON -DARROW_JSON=ON -DARROW_PARQUET=ON -DARROW_WITH_LZ4=ON -DARROW_WITH_SNAPPY=ON -DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON ..
cmake --build . --target install --config Release
popd
5. Run commands below to compile Pybind code:
pushd arrow\python
set PYARROW_BUNDLE_ARROW_CPP=1
python setup.py build_ext --inplace
```
### Component(s)
C++, Python
Contributor guide
Assessment
This issue has not been assessed yet.