`ctypes` does not work on Windows and has memory corruption

I tried to use ctypes to interface to some C/C++ code as DLL with IronPython on Windows. I got random segfaults and errors in malloc. I am trying to boil it down and moved to a simple project that works with CPython but fails with IronPython2.7 and IronPython3.4-beta.

The project to test with is Python-using-C-CPP-libraries-with-ctypes. Great shout-out to @sol-prog

I am using Windows 10 21H1 OS build 19044.1865 and Visual Studio Community 2022. Detailed environment information at the bottom.

I used the released IronPython versions and also built IronPython3.4 from source (so trying that the same compiler is used for the DLL and IronPython binaries).

When I compile the DLL using MSVC (cbmp.dll) and run it with IronPython3.4 the code does not run due to some internal memory corruption.

PS C:\Users\Sascha\Desktop\working_directory\Python-using-C-CPP-libraries-with-ctypes\using_cpp_libs_from_python> ..\..\ironpython3\bin\Debug\net6.0\ipy.exe .\test_cbmp.py
Error in function: BMP_fill_region at line: 114
The region does not fit in the image!

PS C:\Users\Sascha\Desktop\working_directory\Python-using-C-CPP-libraries-with-ctypes\using_cpp_libs_from_python> ..\..\ironpython3\bin\Debug\net46\ipy.exe .\test_cbmp.py
Error in function: BMP_fill_region at line: 114
The region does not fit in the image!

When running with Python3 the code works just fine and produces valid output.

PS C:\Users\Sascha\Desktop\working_directory\Python-using-C-CPP-libraries-with-ctypes\using_cpp_libs_from_python> python3 .\test_cbmp.py
500 500 3

In cbmp.py I had to modify the loading of the library to:

# Find the library and load it
try:
    cbmp = ctypes.cdll.LoadLibrary(".\cbmp")
except OSError:
    print("Unable to load the system C library")
    sys.exit()

I compiled the DLL in the following way (from the "x64 Native Tools Command Prompt for VS 2022")

c:\Users\Sascha\Desktop\working_directory\Python-using-C-CPP-libraries-with-ctypes\using_cpp_libs_from_python>cl /LD /std:c++17 /EHsc /W3 /DBUILD_CBMP cbmp.cpp
Microsoft (R) C/C++ Optimizing Compiler Version 19.32.31332 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

cbmp.cpp
Microsoft (R) Incremental Linker Version 14.32.31332.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:cbmp.dll
/dll
/implib:cbmp.lib
cbmp.obj
   Creating library cbmp.lib and object cbmp.exp

x86 (32-bit) Test

I also build IronPython x86 and the DLL with x86 to test:

From "x86 Native Tools Command Prompt for VS 2022":

c:\Users\Sascha\Desktop\working_directory\Python-using-C-CPP-libraries-with-ctypes\using_cpp_libs_from_python>cl /LD /std:c++17 /EHsc /W3 /DBUILD_CBMP cbmp.cpp
Microsoft (R) C/C++ Optimizing Compiler Version 19.32.31332 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.

cbmp.cpp
Microsoft (R) Incremental Linker Version 14.32.31332.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:cbmp.dll
/dll
/implib:cbmp.lib
cbmp.obj
   Creating library cbmp.lib and object cbmp.exp

Build Python for x86. Only the net46 version seemed to be build:

PS C:\Users\Sascha\Desktop\working_directory\ironpython3> .\make.ps1 -platform x86 debug

This gives some access violation. The code works with CPython and the DLL seems to be fine. I guess something in ctypes is broken in IronPython?!

Or what are the best/correct compiler parameters I should use to compile the DLL with? To match calling conventions, pointer sizes, et cetera?

PS C:\Users\Sascha\Desktop\working_directory\Python-using-C-CPP-libraries-with-ctypes\using_cpp_libs_from_python> ..\..\ironpython3\bin\Debug\net46\ipy32.exe .\test_cbmp.py

Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at InteropInvoker(IntPtr , Object , Object , Object , Object , Object , Object , Object , Object , Object , Object[] )
   at CallSite.Target(Closure , CallSite , Object , Object , Object , Object , Object , Object , Object , Object , Object , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute10[T0,T1,T2,T3,T4,T5,T6,T7,T8,T9,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2, T3 arg3, T4 arg4, T5 arg5, T6 arg6, T7 arg7, T8 arg8, T9 arg9)
   at Microsoft.Scripting.Interpreter.DynamicInstruction`11.Run(InterpretedFrame frame) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\DLR\Src\Microsoft.Dynamic\Interpreter\Instructions\DynamicInstructions.Generated.cs:line 352
   at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\DLR\Src\Microsoft.Dynamic\Interpreter\Interpreter.cs:line 91
   at Microsoft.Scripting.Interpreter.LightLambda.Run12[T0,T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,TRet](T0 arg0, T1 arg1, T2 arg2, T3 arg3, T4 arg4, T5 arg5, T6 arg6, T7 arg7, T8 arg8, T9 arg9, T10 arg10, T11 arg11) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\DLR\Src\Microsoft.Dynamic\Interpreter\LightLambda.Generated.cs:line 506
   at CallSite.Target(Closure , CallSite , CodeContext , Object , Object , Object , Object , Object , Object , Object , Object , Object , Object )
   at Microsoft.Scripting.Interpreter.DynamicInstruction`12.Run(InterpretedFrame frame) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\DLR\Src\Microsoft.Dynamic\Interpreter\Instructions\DynamicInstructions.Generated.cs:line 377
   at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\DLR\Src\Microsoft.Dynamic\Interpreter\Interpreter.cs:line 91
   at Microsoft.Scripting.Interpreter.LightLambda.Run1[T0,TRet](T0 arg0) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\DLR\Src\Microsoft.Dynamic\Interpreter\LightLambda.Generated.cs:line 55
   at IronPython.Compiler.RuntimeScriptCode.InvokeTarget(Scope scope) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\IronPython\Compiler\RuntimeScriptCode.cs:line 64
   at IronPython.Compiler.RuntimeScriptCode.Run(Scope scope) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\IronPython\Compiler\RuntimeScriptCode.cs:line 44
   at IronPython.Hosting.PythonCommandLine.RunFileWorker(String fileName) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\IronPython\Hosting\PythonCommandLine.cs:line 603
   at IronPython.Hosting.PythonCommandLine.RunFile(String fileName) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\IronPython\Hosting\PythonCommandLine.cs:line 558
   at Microsoft.Scripting.Hosting.Shell.CommandLine.Run() in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\DLR\Src\Microsoft.Dynamic\Hosting\Shell\CommandLine.cs:line 132
   at IronPython.Hosting.PythonCommandLine.Run() in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\IronPython\Hosting\PythonCommandLine.cs:line 113
   at Microsoft.Scripting.Hosting.Shell.CommandLine.Run(ScriptEngine engine, IConsole console, ConsoleOptions options) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\DLR\Src\Microsoft.Dynamic\Hosting\Shell\CommandLine.cs:line 100
   at Microsoft.Scripting.Hosting.Shell.ConsoleHost.RunCommandLine() in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\DLR\Src\Microsoft.Dynamic\Hosting\Shell\ConsoleHost.cs:line 384
   at Microsoft.Scripting.Hosting.Shell.ConsoleHost.ExecuteInternal() in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\DLR\Src\Microsoft.Dynamic\Hosting\Shell\ConsoleHost.cs:line 319
   at PythonConsoleHost.ExecuteInternal() in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\IronPythonConsole\Console.cs:line 165
   at Microsoft.Scripting.Hosting.Shell.ConsoleHost.Execute() in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\DLR\Src\Microsoft.Dynamic\Hosting\Shell\ConsoleHost.cs:line 297
   at Microsoft.Scripting.Hosting.Shell.ConsoleHost.Run(String[] args) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\DLR\Src\Microsoft.Dynamic\Hosting\Shell\ConsoleHost.cs:line 197
   at PythonConsoleHost.Main(String[] args) in C:\Users\Sascha\Desktop\working_directory\ironpython3\Src\IronPythonConsole\Console.cs:line 199

Environment

PS C:\Users\Sascha\Desktop\working_directory\ironpython3> .\make.ps1 debug

Welcome to .NET 6.0!
---------------------
SDK Version: 6.0.302

PS C:\Users\Sascha\Desktop\working_directory\ironpython3> dotnet --info
.NET SDK (reflecting any global.json):
 Version:   6.0.302
 Commit:    c857713418

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19044
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.302\

global.json file:
  Not found

Host:
  Version:      6.0.7
  Architecture: x64
  Commit:       0ec02c8c96

.NET SDKs installed:
  6.0.302 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

PS C:\Users\Sascha\Desktop\working_directory\ironpython3> .\bin\Debug\net6.0\ipy.exe
IronPython 3.4.0b1 DEBUG (3.4.0.0010)
[.NETCoreApp,Version=v6.0 on .NET 6.0.7 (64-bit)] on win32
Type "help", "copyright", "credits" or "license" for more information.

PS C:\Users\Sascha\Desktop\working_directory\Python-using-C-CPP-libraries-with-ctypes\using_cpp_libs_from_python> ..\..\ironpython3\bin\Debug\net6.0\ipy.exe -m sysconfig
Platform: "win32"
Python version: "3.4"
Current installation scheme: "nt"

Paths:
        data = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0"
        include = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0\Include"
        platinclude = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0\Include"
        platlib = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0\Lib\site-packages"
        platstdlib = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0\Lib"
        purelib = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0\Lib\site-packages"
        scripts = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0\Scripts"
        stdlib = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0\Lib"

Variables:
        BINDIR = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0"
        BINLIBDEST = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0\Lib"
        EXE = ".exe"
        EXT_SUFFIX = ".pyd"
        INCLUDEPY = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0\Include"
        LIBDEST = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0\Lib"
        SO = ".pyd"
        VERSION = "34"
        abiflags = ""
        base = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0"
        exec_prefix = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0"
        implementation = "IronPython"
        implementation_lower = "ironpython"
        installed_base = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0"
        installed_platbase = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0"
        platbase = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0"
        prefix = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0"
        projectbase = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0"
        py_version = "3.4.0b1"
        py_version_nodot = "34"
        py_version_short = "3.4"
        srcdir = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net6.0"
        userbase = "C:\Users\Sascha\AppData\Roaming\Python"

PS C:\Users\Sascha\Desktop\working_directory\Python-using-C-CPP-libraries-with-ctypes\using_cpp_libs_from_python> ..\..\ironpython3\bin\Debug\net46\ipy.exe
IronPython 3.4.0b1 DEBUG (3.4.0.0010)
[.NETFramework,Version=v4.6 on .NET Framework 4.8.4515.0 (64-bit)] on win32

PS C:\Users\Sascha\Desktop\working_directory\Python-using-C-CPP-libraries-with-ctypes\using_cpp_libs_from_python> ..\..\ironpython3\bin\Debug\net46\ipy.exe -m sysconfig
Platform: "win32"
Python version: "3.4"
Current installation scheme: "nt"

Paths:
        data = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46"
        include = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46\Include"
        platinclude = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46\Include"
        platlib = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46\Lib\site-packages"
        platstdlib = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46\Lib"
        purelib = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46\Lib\site-packages"
        scripts = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46\Scripts"
        stdlib = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46\Lib"

Variables:
        BINDIR = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46"
        BINLIBDEST = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46\Lib"
        EXE = ".exe"
        EXT_SUFFIX = ".pyd"
        INCLUDEPY = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46\Include"
        LIBDEST = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46\Lib"
        SO = ".pyd"
        VERSION = "34"
        abiflags = ""
        base = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46"
        exec_prefix = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46"
        implementation = "IronPython"
        implementation_lower = "ironpython"
        installed_base = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46"
        installed_platbase = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46"
        platbase = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46"
        prefix = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46"
        projectbase = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46"
        py_version = "3.4.0b1"
        py_version_nodot = "34"
        py_version_short = "3.4"
        srcdir = "C:\Users\Sascha\Desktop\working_directory\ironpython3\bin\Debug\net46"
        userbase = "C:\Users\Sascha\AppData\Roaming\Python"

"x64 Native Tools Command Prompt for VS 2022"

c:\Users\Sascha\Desktop\working_directory\Python-using-C-CPP-libraries-with-ctypes\using_cpp_libs_from_python>cl.exe
Microsoft (R) C/C++ Optimizing Compiler Version 19.32.31332 for x64

PS C:\Users\Sascha\Desktop\working_directory\Python-using-C-CPP-libraries-with-ctypes\using_cpp_libs_from_python> python3 -m sysconfig
Platform: "win-amd64"
Python version: "3.10"
Current installation scheme: "nt"

Paths:
        data = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0"
        include = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\Include"
        platinclude = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\Include"
        platlib = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\Lib\site-packages"
        platstdlib = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\Lib"
        purelib = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\Lib\site-packages"
        scripts = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\Scripts"
        stdlib = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\Lib"

Variables:
        BINDIR = "C:\Users\Sascha\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0"
        BINLIBDEST = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\Lib"
        EXE = ".exe"
        EXT_SUFFIX = ".cp310-win_amd64.pyd"
        INCLUDEPY = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\Include"
        LIBDEST = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\Lib"
        SO = ".cp310-win_amd64.pyd"
        TZPATH = ""
        VERSION = "310"
        abiflags = ""
        base = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0"
        exec_prefix = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0"
        installed_base = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0"
        installed_platbase = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0"
        platbase = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0"
        platlibdir = "lib"
        prefix = "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0"
        projectbase = "C:\Users\Sascha\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0"
        py_version = "3.10.6"
        py_version_nodot = "310"
        py_version_nodot_plat = "310"
        py_version_short = "3.10"
        srcdir = "C:\Users\Sascha\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0"
        userbase = "C:\Users\Sascha\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages"

Start by reproducing the failure with test_cbmp.py and the cbmp.cpp DLL under CPython and the IronPython builds described. Trace the ctypes call through the InteropInvoker stack, then verify that the same example completes without memory corruption and produces the expected 500 500 3 output.