microsoft / microsoft/onnxruntime
Cuda allocator pointer alignment
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 8h
- Merged PRs (30d)
- 179
Description
Describe the bug
Cuda allocator returns pointers that are not aligned correctly. When I read cudaDeviceProp::textureAlignment, the value is 512, but Ort::Allocator::Alloc() returns pointers aligned to 256. I initialize Ort::Allocator using Ort::MemoryInfo("Cuda", OrtAllocatorType::OrtArenaAllocator, deviceIndex, OrtMemTypeDefault), but I've tried OrtAllocatorType::OrtDefaultAllocator and it does not change the alignment. If I allocate some memory using cudaMalloc I always get pointers aligned to 512.
This alignment issue prevents me from creating a Cuda texture because pointer must be aligned properly. If I just run kernels on that memory, it runs fine, just texture creation is a problem.
Urgency
None
System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
- ONNX Runtime installed from (source or binary): binary
- ONNX Runtime version: 1.10
- Python version: 3.8
- Visual Studio version (if applicable): 2019
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version: 10.2
- GPU model and memory: 2080 Ti, 11GB
To Reproduce
- Create Ort::MemoryInfo object with arguments ("Cuda", OrtAllocatorType::OrtArenaAllocator, deviceIndex, OrtMemTypeDefault)
- Create Ort::Allocator object with session with these provider options changed:
OrtCUDAProviderOptions providerOptions;
providerOptions.device_id = DeviceIndex;
providerOptions.has_user_compute_stream = 1;
providerOptions.user_compute_stream = 0; - Run Ort::Allocator with 1048576 bytes
- I get pointer aligned to 256
- When I call cudaTextureCreate it sometimes fails because pointer is not aligned to 512
- I have tried this with many different models. Behavior is that for some it will be well aligned, by chance, and for some it won't. If I add a few more dummy allocations, it eventually does align with 512 and the texture creation works.
Expected behavior
I expect Ort::Allocator to provide arrays that are well aligned, so that output can be used to create a Cuda texture.
Screenshots
None
Additional context
None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue through Ort::MemoryInfo, Ort::Allocator::Alloc(), and the CUDA provider options described in the report, then compare the result with cudaMalloc and cudaDeviceProp::textureAlignment. Investigate the CUDA allocator entry point and any existing allocator tests. Done means allocations consistently meet the required 512-byte alignment and can be used by cudaTextureCreate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100