microsoft / microsoft/onnxruntime

initial_chunk_size_bytes is a signed int (limited to 2 GiB)

Open
#23,023 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

For use with multiple sessions and a shared memory arena, where I know that each session will use no more than X GiB, [the documentation](https://onnxruntime.ai/docs/get-started/with-c.html) suggests setting `initial_chunk_size_bytes` such that most model requests will be completed without needing to allocate more memory.

Unfortunately, `initial_chunk_size_bytes` is a signed int ([struct definition here](https://github.com/microsoft/onnxruntime/blob/6ed77cc374c04956bc1197d0dc0fe5a2ed9b15b9/include/onnxruntime/core/framework/allocator.h#L17-L40)), and thus has a maximum value of `2**31 - 1` or just under 2 GiB. In my case, I know that my models will never use more than 4 GiB during inference, but I cannot set this, as it is too large.

Is there a reason not to use `int64_t` for this?

### To reproduce

This is reproducible in either C++ or Python. Here's the Python:
```python3
import onnxruntime

# This works:
arena_cfg = onnxruntime.OrtArenaCfg({'initial_chunk_size_bytes': 2 * 1024 * 1024 * 1024 - 1})

# This throws an exception:
arena_cfg = onnxruntime.OrtArenaCfg({'initial_chunk_size_bytes': 2 * 1024 * 1024 * 1024})
```

### Urgency

_No response_

### Platform

Linux

### OS Version

Ubuntu 22

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.17.0

### ONNX Runtime API

Python

### Architecture

X64

### Execution Provider

CUDA

### Execution Provider Library Version

_No response_

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The allocator configuration struct is linked in include/onnxruntime/core/framework/allocator.h; start by inspecting initial_chunk_size_bytes and tracing the Python OrtArenaCfg binding. Reproduce the 2 GiB boundary from the issue, then verify that a 4 GiB value is accepted without the reported exception in the affected APIs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
api, backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.