Model.to_gpu is not usable
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 295
- PR merge metrics
- No merged PRs in 30d
Description
I am attempting to assign individual layers to separate GPUs in order to conserve memory. However, the Model.to_gpu function takes an all or nothing approach which prevents this from working.
While diagnosing the origin of memory access error during training, (`cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered`), I noticed that [`CupyOps.device_id`](https://github.com/explosion/thinc/blob/master/thinc/backends/cupy_ops.py#L23) is never used
[or set](https://github.com/explosion/thinc/blob/c7b0d6759645babe94315a36c84d56ec877252f2/thinc/model.py#L513).
Ideally, all the CupyOps would run inside a `cp.cuda.Device(device_id)` context, but that is not the case. Instead, the `xp` attribute is (ab)used in many places. That will try and run everything through GPU 0, so errors won't appear until something was moved to another GPU.
Two other difficulties are the initialization step, which doesn't declare memory in the right places,
and the `finish_update` step, where the optimizer does arithmetic on parameters outside of a context.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.