microsoft / microsoft/onnxruntime

Shared Session Allocator Causes Crash in Layer

Open
#19,795 7 comments 0 reactions 1 assignee Claimed by @pranavsharma View on GitHub
core runtime more info needed
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

[In onnxruntime\onnxruntime\core\mlas\lib\sgemm.cpp](https://github.com/microsoft/onnxruntime/blob/e93a860819545ea64acfe36e19e2b954389d48bf/onnxruntime/core/mlas/lib/sgemm.cpp#L1065) throws access violation reading location 0xFFFF... when enabled shared session usage of a custom allocator kOrtSessionOptionsConfigUseEnvAllocators. This happens even when only a single session has been created.

The error itself seems to be happening in MlasGemmFloatKernalFma3, but I don't have the symbols loaded for that (any help there would be appreciated, I've custom built, and supposedly enabled all debug functionality).

I am following https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/test/shared_lib/test_inference.cc, and I believe I'm following it exactly. One thing I may be getting wrong is that a different MockedAllocator instance is being used for initializing the tensors in the example, I'm not sure why this is important. I tried this and same results, though.

I have confirmed that my custom onnx build passes the test.

### To reproduce

Here is a minimal example -

Custom allocator:

```c++
struct CustomAllocator: public OrtAllocator {
private:
const OrtApi* m_ort;
OrtMemoryInfo* memory_info;
public:
CustomAllocator(const OrtApi* ort) : m_ort{ ort } {
OrtAllocator::version = ORT_API_VERSION;
OrtAllocator::Alloc = [](OrtAllocator* this_, size_t size) {
return static_cast(this_)->Alloc(size);
};
OrtAllocator::Free = [](OrtAllocator* this_, void* p) { static_cast(this_)->Free(p); };
OrtAllocator::Info = [](const OrtAllocator* this_) { return static_cast(this_)->Info(); };
ORT_ABORT_ON_ERROR(m_ort->CreateCpuMemoryInfo(OrtDeviceAllocator, OrtMemTypeDefault, &memory_info));
}
void Release() {
m_ort->ReleaseMemoryInfo(memory_info);
}
void* Alloc(size_t size) {
return ptr = ::malloc(size);
}
void Free(void* p) {
::free(p);
}
const OrtMemoryInfo* Info() const {
return memory_info;
}
};
```

Env creation:
```c++
OrtThreadingOptions* tpOptions;
ORT_ABORT_ON_ERROR(m_ort->CreateThreadingOptions(&tpOptions));
ORT_ABORT_ON_ERROR(m_ort->SetGlobalInterOpNumThreads(tpOptions, 1));
ORT_ABORT_ON_ERROR(m_ort->SetGlobalIntraOpNumThreads(tpOptions, 1));
ORT_ABORT_ON_ERROR(m_ort->SetGlobalSpinControl(tpOptions, 0));
ORT_ABORT_ON_ERROR(m_ort->CreateEnvWithGlobalThreadPools(ORT_LOGGING_LEVEL_VERBOSE, "test", tpOptions, &m_env));
m_allocator = new CustomAllocator(m_ort);
ORT_ABORT_ON_ERROR(m_ort->RegisterAllocator(m_env, m_allocator));
```

Later... session creation and usage

```c++
// session creation
ORT_ABORT_ON_ERROR(m_ort->CreateSessionOptions(&m_session_options));
m_ort->AddSessionConfigEntry(m_session_options, kOrtSessionOptionsConfigUseEnvAllocators, "1");
m_ort->DisableCpuMemArena(m_session_options);
m_ort->DisablePerSessionThreads(m_session_options);
ORT_ABORT_ON_ERROR(m_ort->CreateSessionFromArray(m_env data, size, m_session_options, &m_session));

// inference
size_t in_dims[2] = {1, this->m_model.input_len};
size_t out_dims[2] = {1, this->m_model.output_len};
OrtValue* in_tensor;
int* mutable_in_storage;
ORT_ABORT_ON_ERROR(m_ort->CreateTensorAsOrtValue(m_allocator, (const int64_t*)in_dims, 2, TypeToTensorType::type, &in_tensor));
ORT_ABORT_ON_ERROR(m_ort->GetTensorMutableData(in_tensor, reinterpret_cast(&mutable_in_storage)));
memcpy(mutable_in_storage, in_elements.data(), this->m_model.input_len * sizeof(int));

OrtValue* out_tensor;
float* mutable_out_storage;
ORT_ABORT_ON_ERROR(m_ort->CreateTensorAsOrtValue(m_allocator, (const int64_t*)out_dims, 2, TypeToTensorType::type, &out_tensor));
ORT_ABORT_ON_ERROR(m_ort->GetTensorMutableData(in_tensor, reinterpret_cast(&mutable_out_storage)));
memcpy(mutable_out_storage, out_elements.data(), this->m_model.output_len * sizeof(float));

ORT_ABORT_ON_ERROR(m_ort->Run(m_session, NULL, &input_name, &in_tensor, 1, &output_name, 1, &out_tensor));
```

### Urgency

_No response_

### Platform

Windows

### OS Version

10

### ONNX Runtime Installation

Built from Source

### ONNX Runtime Version or Commit ID

rel-1.16.3

### ONNX Runtime API

C

### Architecture

X64

### Execution Provider

Default CPU

### Execution Provider Library Version

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.