[Bug] Concurrent access to a shared PythonInterpreter can crash the JVM in async cross-language calls
- Dominant language
- Java
- Stars
- 452
- Forks
- 167
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 49
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.
### Description
The embedded `pemja.core.PythonInterpreter` owned by an operator is shared by cross-language action and resource paths. When multiple threads concurrently enter the same interpreter through Pemja APIs such as `invoke`, the JVM can terminate with a native `SIGSEGV` instead of returning a managed Java or Python exception.
This is broader than the Mem0 path where the problem was first observed. Any asynchronous cross-language path that can concurrently access the same interpreter is potentially affected. The participating threads may originate from:
- Flink's Java async executor;
- the mailbox or lifecycle thread;
- Python-created workers that call into Java and then re-enter the same interpreter through a Java adapter; or
- another JVM thread carrying a cross-language callback.
This does not mean that every asynchronous operation is unsafe. A Python worker that only runs Python code, a callback that only executes Java code without touching the interpreter, sequential cross-language calls, and calls using different interpreter instances are outside the demonstrated failure condition.
The failure was first observed in the cross-language Mem0 long-term-memory E2E test in this CI job:
https://github.com/apache/flink-agents/actions/runs/33615851952/job/100201365522
The original failing path was:
```text
Java durable async worker
-> Java MemorySet.add
-> Java Mem0LongTermMemory wrapper
-> shared Pemja PythonInterpreter.invoke
-> Python Mem0LongTermMemory.add
-> Mem0 chat / embedding / vector-store processing
```
A sanitized excerpt from a local reproduction is:
```text
# A fatal error has been detected by the Java Runtime Environment:
# SIGSEGV (0xb)
# JRE version: Java(TM) SE Runtime Environment (21.0.8+12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (bsd-aarch64)
Current thread: JavaThread "flink-agents-java-async-pool-5-thread-2" [_thread_in_native]
C [Python] _PyFrame_Clear
C [Python] _PyEval_EvalFrameDefault
C [pemja_core.cpython-311-darwin.so] JcpPyObject_Call
C [pemja_core.cpython-311-darwin.so] Java_pemja_core_PythonInterpreter_invoke
j pemja.core.PythonInterpreter.invoke(...)
j org.apache.flink.agents.runtime.python.utils.PythonResourceAdapterImpl.callMethod(...)
```
### Root-cause isolation
The failure was isolated from Mem0 with a ChatModel cross-language E2E test:
1. Configure `AgentExecutionOptions.NUM_ASYNC_THREADS` to `2`.
2. Send two events with different keys so that two slow chat requests overlap.
3. Use a Java agent calling a Python ChatModel setup through the shared interpreter.
Both connection variants are affected:
- Java -> Python setup -> Java connection; and
- Java -> Python setup -> Python connection.
In the Python-connection variant, the JVM crashed before the Python connection wrapper's `chat()` entry log was emitted. The fatal stack was already inside `PythonResourceAdapterImpl.callMethod -> PythonInterpreter.invoke`, which excludes the connection implementation, Mem0's internal executor, and Elasticsearch as required causes.
Expected behavior: asynchronous cross-language execution completes safely, or an unsupported concurrent access pattern fails with a managed exception without terminating the JVM.
Actual behavior: concurrent access to the shared interpreter can terminate the entire JVM in CPython evaluation through Pemja.
### How to reproduce
#### Isolated ChatModel reproduction
1. Build Flink Agents and install its Python dependencies.
2. Start Ollama with a chat model such as `qwen3:1.7b`.
3. Configure the cross-language ChatModel test with two async threads and two different input keys.
4. Use a Python ChatModel setup backed by either a Java or Python connection.
5. Make the two requests slow enough to overlap and run `ChatModelCrossLanguageTest`.
When the requests overlap, the Surefire JVM may abort before both requests complete and produce an `hs_err_pid*.log`.
#### Original Mem0 reproduction
1. Start Ollama with the `nomic-embed-text` model.
2. Start Elasticsearch and export `ES_HOST=http://localhost:9200`.
3. Export a valid `ACTION_API_KEY` and, if needed, `ACTION_BASE_URL` for the OpenAI-compatible chat model.
4. Run:
```bash
mvn -pl e2e-test/flink-agents-end-to-end-tests-resource-cross-language \
-Dtest=Mem0LongTermMemoryTest test
```
The test sends interleaved facts for two keys. `Mem0LongTermMemoryAgent.addItems()` invokes `memorySet.add()` using `durableExecuteAsync()`. One of the overlapping additions may abort the Maven Surefire JVM and produce an `hs_err_pid*.log`.
### Version and environment
- Flink Agents: `0.4-SNAPSHOT`
- Flink: `2.3.0`
- Pemja: `0.5.7`
- CI: `ubuntu-latest`, Temurin JDK 21, Python 3.12
- Local reproduction: macOS aarch64, Oracle JDK 21.0.8, Python 3.11
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
Assessment
This issue has not been assessed yet.