microsoft / microsoft/onnxruntime-inference-examples
Ort::MemoryInfo with CUDA in c++?
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.7k
- Forks
- 414
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 14
Description
Hello,
I'm trying bind some output values to CUDA to avoid copying back to CPU.
I want to do the c++ equivalent of
io.bind_output(name, 'cuda')
and in the following frames, bind the input to the value from the output of the previous frame.
I guess I should do it by the function BindOutput and configure memoryInfo to use CUDA memory.
io_binding.BindOutput(name, memoryInfo);
But I'm not sure how to configure memoryInfo to use CUDA memory because I couldn't find any example doing it.
By searching on other issues, I saw some code saying it's not possible (unless added recently)
//as of 4th Feb 2022 Onnx only supports allocation on the CPU
Ort::MemoryInfo memoryInfo = Ort::MemoryInfo::CreateCpu(
OrtAllocatorType::OrtArenaAllocator, OrtMemType::OrtMemTypeDefault);
but also saw some code that seems to do it :
Ort::MemoryInfo memoryInfo("Cuda", OrtAllocatorType::OrtDeviceAllocator, 0, OrtMemType::OrtMemTypeDefault);
Due to the lack of documentation, I could not be sure if this second code is really setting for CUDA memory without CPU copy or not.
Does the parameter name ("Cuda") corresponds to the type of memory or is it just a name we give to the memoryInfo object without effect? Does OrtDeviceAllocator means GPU/CUDA memory? Is OrtMemTypeDefault the good value?
Can anyone confirm it?
Many thanks
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the C++ BindOutput and Ort::MemoryInfo APIs referenced in the report, comparing them with the Python io.bind_output example. Done means documenting whether the shown CUDA configuration is supported and which allocator and memory type values are valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100