apache / apache/flink-agents

[Bug] Pemja reference leaks in action execution and resource bridges

Closed
#1,057 0 comments 0 reactions 1 assignee Claimed by @joeyutong View on GitHub
affectVersion/0.2.1 affectVersion/0.3.1 bug priority/major
Dominant language
Java
Stars
452
Forks
167
Avg merge
5d 9h
Merged PRs (30d)
49

Description

### Search before asking

- [x] I searched existing issues.

Related Action report: #1048. Fix: #1058.

### Description

Repeated Python Action execution and resource calls create Pemja `PyObject` handles that own native Python references. Dropping the Java local variable does not release that reference; [`PyObject.close()`](https://github.com/alibaba/pemja/blob/release-0.5.7/src/main/java/pemja/core/object/PyObject.java#L20-L54) performs the native decref. Handles left unclosed can retain Events, coroutines, messages, documents, and resource objects after Java has finished using them.

The affected lifetimes are:

- **Actions:** converted Events, returned coroutine wrappers, wrappers retrieved for each poll, ignored yielded/returned values, and interpreter globals left after completion.
- **Resource calls:** temporary ChatMessage/tool wrappers, input/output Documents and normalized vector queries, formatted MCP messages, and Mem0 MemorySet/MemorySetItem wrappers.
- **Resource shutdown:** chat/embedding connection and setup roots, vector-store roots, and MCP Server/Tool/Prompt roots. MCP discovery registers tools and prompts but leaves the Server wrapper outside the resource cache's shutdown path.

The Python connection's logical `close()` and the Java `PyObject` handle's native-reference release are separate operations. Calling the former through a Tool or Prompt does not release the Server wrapper's own handle.

This issue covers temporary values consumed by framework calls and root handles released during normal resource shutdown. Resource construction/initialization/discovery failure cleanup and ownership of arbitrary outgoing Tool results or custom metadata values are outside its scope.

### Observed behavior

Local Flink reproductions show native-reference retention after forced Java/Python garbage collection:

| Path | Workload | Retention when the corresponding reference is not released |
|---|---|---|
| Converted Action Event | 100 records, 1 MiB payload each | 100 Events and 100 MiB of payload |
| Coroutine wrapper returned by an Action or retrieved for polling | 100 records, 1 MiB payload each; each boundary measured separately | 100 Events and 100 awaitables retaining 100 MiB |
| Completed-coroutine global | 100 completed Actions | 100 globals and 100 awaitables |
| Ignored yielded or returned output | 100 records, 1 MiB payload each; each output path measured separately | 100 output objects and 100 MiB |
| Chat bridge | 200 iterations, 256 KiB per tracked object | 600 messages, 200 tool values, and 200 MiB |
| Vector bridge | 200 iterations, 256 KiB per tracked object | 600 documents, 200 normalized handles, and about 100 MiB |
| Resource recreation | 200 resources, 256 KiB each | 200 resources and 50 MiB; no logical close calls |

Each ownership boundary was measured in a fresh TaskManager. Live-object counts and reachable payload bytes are the evidence here; these are bug-reproduction observations, not memory measurements of PR #1058.

### How to reproduce

1. Use an affected build to repeatedly execute Python Actions or invoke Python-backed chat/vector resources in a long-lived interpreter.
2. Give tracked Python objects a measurable payload and keep weak references to them. For resource recreation, also count logical close calls.
3. Allow each operation to complete, then force Java/Python garbage collection and count surviving objects. For async Actions, count completed-coroutine globals as well.
4. Exercise the ownership boundaries separately to distinguish an unclosed Java handle from an interpreter-global reference.

Expected behavior:

- Temporary handles are released after use or conversion, including nested handles in a wholly discarded Action output.
- A pending coroutine remains alive and resumable through its interpreter-global reference; completion removes that global.
- `call_python_awaitable` retains its `(finished, yielded_or_returned_value)` contract.
- Normal resource shutdown invokes Python `close()` and releases each wrapper's native handle once, including the MCP Server wrapper used for discovery.
- Outgoing Tool results and custom metadata values preserve their existing representation and ownership.

### Version and environment

- Reported deployment: Flink Agents `release-0.3`.
- Local reproduction: Flink 2.2.0, Pemja 0.5.7, Java 17, CPython 3.11.15, macOS.
- Implementation and verification: #1058.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.