microsoft / microsoft/onnxruntime

Cuda allocator pointer alignment

Open
#12,289 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

core runtime feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 8h
Merged PRs (30d)
179

Description

Describe the bug
Cuda allocator returns pointers that are not aligned correctly. When I read cudaDeviceProp::textureAlignment, the value is 512, but Ort::Allocator::Alloc() returns pointers aligned to 256. I initialize Ort::Allocator using Ort::MemoryInfo("Cuda", OrtAllocatorType::OrtArenaAllocator, deviceIndex, OrtMemTypeDefault), but I've tried OrtAllocatorType::OrtDefaultAllocator and it does not change the alignment. If I allocate some memory using cudaMalloc I always get pointers aligned to 512.

This alignment issue prevents me from creating a Cuda texture because pointer must be aligned properly. If I just run kernels on that memory, it runs fine, just texture creation is a problem.

Urgency
None

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
  • ONNX Runtime installed from (source or binary): binary
  • ONNX Runtime version: 1.10
  • Python version: 3.8
  • Visual Studio version (if applicable): 2019
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version: 10.2
  • GPU model and memory: 2080 Ti, 11GB

To Reproduce

  • Create Ort::MemoryInfo object with arguments ("Cuda", OrtAllocatorType::OrtArenaAllocator, deviceIndex, OrtMemTypeDefault)
  • Create Ort::Allocator object with session with these provider options changed:
    OrtCUDAProviderOptions providerOptions;
    providerOptions.device_id = DeviceIndex;
    providerOptions.has_user_compute_stream = 1;
    providerOptions.user_compute_stream = 0;
  • Run Ort::Allocator with 1048576 bytes
  • I get pointer aligned to 256
  • When I call cudaTextureCreate it sometimes fails because pointer is not aligned to 512
  • I have tried this with many different models. Behavior is that for some it will be well aligned, by chance, and for some it won't. If I add a few more dummy allocations, it eventually does align with 512 and the texture creation works.

Expected behavior
I expect Ort::Allocator to provide arrays that are well aligned, so that output can be used to create a Cuda texture.

Screenshots
None

Additional context
None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue through Ort::MemoryInfo, Ort::Allocator::Alloc(), and the CUDA provider options described in the report, then compare the result with cudaMalloc and cudaDeviceProp::textureAlignment. Investigate the CUDA allocator entry point and any existing allocator tests. Done means allocations consistently meet the required 512-byte alignment and can be used by cudaTextureCreate.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.