explosion / explosion/thinc

Model.to_gpu is not usable

Open
#713 3 comments 0 reactions 0 assignees View on GitHub
feat / ops
Dominant language
Python
Stars
2.9k
Forks
295
PR merge metrics
No merged PRs in 30d

Description

I am attempting to assign individual layers to separate GPUs in order to conserve memory. However, the Model.to_gpu function takes an all or nothing approach which prevents this from working.

While diagnosing the origin of memory access error during training, (`cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered`), I noticed that [`CupyOps.device_id`](https://github.com/explosion/thinc/blob/master/thinc/backends/cupy_ops.py#L23) is never used
[or set](https://github.com/explosion/thinc/blob/c7b0d6759645babe94315a36c84d56ec877252f2/thinc/model.py#L513).

Ideally, all the CupyOps would run inside a `cp.cuda.Device(device_id)` context, but that is not the case. Instead, the `xp` attribute is (ab)used in many places. That will try and run everything through GPU 0, so errors won't appear until something was moved to another GPU.

Two other difficulties are the initialization step, which doesn't declare memory in the right places,
and the `finish_update` step, where the optimizer does arithmetic on parameters outside of a context.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.