huggingface / huggingface/candle

Transfering CUDA tensors between candle and tch-rs without synchronization

Open
#973 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Is there a way to transfer CUDA tensors between candle and tch-rs without requiring GPU/CPU synchronisation?

I am doing reinforcement learning and currently I have a ResNet that I am training in tch-rs. I have found that the main bottleneck of my implementation is GPU/CPU synchronization when creating observations of the environment. To avoid this I want to try using candle to reimplement the environment to keep everything on the GPU. Before I can do that I need to figure out whether I can send CUDA tensors from candle to a network implemented in tch-rs.

---

Related questions that you may also be able to answer: How would I send tensors between candle and pytorch+jax? What about tch-rs and pytorch+jax?

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Start by investigating the tensor interoperability APIs for candle, tch-rs, CUDA, PyTorch, and JAX; done would require a clearly documented, supported way to transfer tensors without GPU/CPU synchronization, if one is feasible.

Written by the indexing model from the issue text.

Assessment

Tech stack
pytorch, rust
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.