How to properly hold onto an object in embedded mode?

Hello! First, I'd like to thank you for the amazing work on this library. It's a mind-bending level of inter-language interaction, IMO. 👏

I was experimenting with it in the embedded mode. Not sure if I was using it in the intended way, but... I took a considerably sized web-app and tried to add some kind of a debugging/introspection interface to it with clj & cljs.

The app uses gevent internally for non-blocking IO operations, so the setup was a bit involving, but in the end it worked out. I'll explain it to get a better picture:

• app starts as a Python process
• gevent is called at the earliest to do its monkey-patching of IO
• python app continues it's startup
• separate python thread (real OS thread) is started, which does all JVM/clojure initialization (one more thread I guess) and than calls -main of the clojure's part of the app
• in the main thread of the python app a queue is created and worker greenlet (fake thread or coroutine) is started to watch it
• when clojure world needs to call something from python world, that involves gevent, it puts a task into a queue for the worker to execute it on the main thread
• if task needs to return something to clojure world, a one-off queue is created to transfer the return value

It worked fine, I've even made a little introspection tool, that can peer into a running python app vars and call functions, toggle stuff and so on.

Then, I've tried to implement a tap> like system (just putting python objects into an atom for later inspection), to send info from python to clojure... And was abruptly stopped by a SIGSEGV. :(

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000aaaac541ef74, pid=1685, tid=1869
#
# JRE version: OpenJDK Runtime Environment (17.0.12+7) (build 17.0.12+7-Ubuntu-1ubuntu222.04)
# Java VM: OpenJDK 64-Bit Server VM (17.0.12+7-Ubuntu-1ubuntu222.04, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# C  [python3+0xcef74]
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /var/www/hs_err_pid1685.log
#
# If you would like to submit a bug report, please visit:
#   https://bugs.launchpad.net/ubuntu/+source/openjdk-17
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

It seems that objects I send through "tap" got GC'ed on python part. I've tried to find something about holding to objects in the documentation, but nothing was particularly fitting to my case. Digging through the code, I've stumbled upon track-pyobject and incref-and-track. Using the latter on the objects I send to clojure world did not help though... I still got a SIGSEGV, but in different internal python function. To validate my assumption about objects being GD'ed, I've implemented a dumb "borrowing" mechanics on python side (putting objects, to be sent to clojure, in a dict) an it worked okay.

I'd prefer not to do ->jvm on objects, because the idea is to use them again in a python app to replay some actions that it performs.

My question: Is there a way to hold onto python objects from clojure in embedded mode so that refcounts will be decreased when references to the pyobjs are GC'ed on the JVM?

Thank you for your time :)

Python 3.10.12
Clojure 1.12
openjdk 17.0.12 2024-07-16
OpenJDK Runtime Environment (build 17.0.12+7-Ubuntu-1ubuntu222.04)
OpenJDK 64-Bit Server VM (build 17.0.12+7-Ubuntu-1ubuntu222.04, mixed mode, sharing)

O relatório menciona o modo incorporado e os pontos de entrada track-pyobject e incref-and-track; comece rastreando a propriedade e o comportamento do refcount deles enquanto reproduz o SIGSEGV com Python 3.10 e Clojure 1.12. O trabalho estará concluído quando for estabelecida uma forma segura de manter objetos Python durante a coleta de lixo da JVM, ou quando for documentado que o modo incorporado não oferece suporte a esse uso.