JuliaPy / JuliaPy/pyjulia

Ctypes ReadOnlyMemoryError

Open
#305 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
893
Forks
101
PR merge metrics
No merged PRs in 30d

Description

I am currently running into an issue using pyjulia. I am performing a nonlinear optimization of an objective function that requires some heavy numerical integration within the objective function. When it was in pure Python, I had it working just fine, but it was understandably slow, and I have switched the numerical integration to Julia.

The overall code is too long to reproduce here, but the general workflow is:

- [nlopt](https://nlopt.readthedocs.io/en/latest/) called from Python
- Within the objective function, numerical integration is performed in Julia and returned to Python
- Within the objective function, Scipy's fsolve() is used to solve a nonlinear system of equations based on the output
- Basic math is performed on the output from fsolve, and returned to nlopt

Without any attempts to debug, this optimizer will iterate for approximately 15 minutes (a couple thousand iterations) before I get a ReadOnlyMemoryError. I've pasted the output at the bottom of this issue.

I've tried a good number of ways of overcoming this error, but only seem to be able to delay it for a time:

- Switching the optimization algorithm within nlopt. No change.
- Importing julia within the "master" python script before any other modules. No change.
- Putting a time.sleep() before and after the julia numerical integration. My thought being that julia may be too fast for python in an iteration like this, and it may be going too fast for the Global Interpreter Lock to be released. It usually takes longer for the error to appear, but it still inevitably shows up, usually after around 30 minutes.
- Performing regular Julia garbage collection `Base.GC.gc()` at some cadence within the iteration. Generally speaking, the more rapid the cadence, the longer it will last before running into the error, but the slower the iteration. Performing garbage collection after every objective function evaluation is the only way I've found to make the optimization last an entire hour (which is what I currently have as the timeout time for the optimizer).
- Combinations of the previous two steps. Time.sleep() and Julia garbage collection. Currently by best attempt is performing garbage collection every 100 iterations and a pause of 0.001 seconds every iteration.

The program isn't using significant RAM, so I'm doubtful of a out of memory issue.

Versions of relevant modules:
Python 3.7.2
Pyjulia 0.4.1
numpy 1.16.4
scipy 1.3.0
nlopt 2.6.1
Windows

Error message:
```
fatal: error thrown and no exception handler available.
ReadOnlyMemoryError()
DllCanUnloadNow at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\DLLs\_ctypes.pyd (unknown line)
unknown function (ip: 00007FFD716C6D79)
PyObject_FastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyFunction_FastCallDict at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallDict at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_FastCallDict at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_CallFunctionObjArgs at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyNumber_Invert at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyArg_UnpackStack at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_GetAttr at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyFunction_FastCallDict at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_Call at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyInit__minpack at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\_minpack.cp37-win_amd64.pyd (unknown line)
PyInit__minpack at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\_minpack.cp37-win_amd64.pyd (unknown line)
fdjac1 at C:\projects\scipy-wheels\scipy\scipy/optimize/minpack\fdjac1.f:112
hybrd at C:\projects\scipy-wheels\scipy\scipy/optimize/minpack\hybrd.f:226
PyInit__minpack at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\_minpack.cp37-win_amd64.pyd (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyFunction_FastCallDict at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_SetAttr at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyFunction_FastCallDict at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_Call at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
unknown function (ip: 00007FFD5CAD2A83)
unknown function (ip: 00007FFD5CAD335B)
unknown function (ip: 00007FFD5B086069)
unknown function (ip: 00007FFD5B085DD9)
nlopt_optimize at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\lib\site-packages\nlopt\nlopt.dll (unknown line)
nlopt_optimize at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\lib\site-packages\nlopt\nlopt.dll (unknown line)
unknown function (ip: 00007FFD5CAD36ED)
PyInit__nlopt at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\lib\site-packages\nlopt\_nlopt.pyd (unknown line)
unknown function (ip: 00007FFD5CAC78C2)
PyErr_NoMemory at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_SetAttr at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeEx at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCode at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyFuture_FromASTObject at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyRun_FileExFlags at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyRun_SimpleFileExFlags at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyRun_AnyFileExFlags at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
Py_UnixMain at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
Py_UnixMain at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyErr_NoMemory at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
Py_Main at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
Py_Main at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
unknown function (ip: 00007FF716F61257)
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.