microsoft / microsoft/onnxruntime-genai

Memory leak/growth in iOS build when pre-filling tokens via generator's `AppendTokenSequences` API

Open
#1,540 1 comment 0 reactions 1 assignee View on GitHub

@baijumeswani is already working on this.

Since Oct 3, 2025.

platform:mobile
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

Describe the bug
We are noticing a memory leak/growth in the iOS build when we load historical chats via the generator's AppendTokenSequences API.

Note that the same code works on the Android side.

Here's the snippet of wrapper code in C++ we have in common to Android/iOS:

void ONNXLLMExecutor::add_input_to_generator(const std::string& input) {
  auto sequences = OgaSequences::Create();
  _tokenizer->Encode(input.c_str(), *sequences);
  _generator->AppendTokenSequences(*sequences);
}

Expected behavior
No memory leak/growth when tokens are pre-filled from a historical chat

Screenshots
N/A

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone 15]
  • OS: [e.g. iOS 18.5, 17.5]
  • Version [v0.7.0 & v0.8.2]

Additional context
Possibly related issue https://github.com/microsoft/onnxruntime-genai/issues/1075 which was auto closed.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.