RuntimeError: CUDA error: an illegal memory access was encountered while setting image for predictor

Hi! I am trying to use Segment Anything, but I constantly get the following error: RuntimeError: CUDA error: an illegal memory access was encountered.

Here is a small piece of my code:

```
# build model
sam = sam_model_registry[model_name](checkpoint=weights_path)
# load model on device
torch.cuda.set_device(0)
torch_device = torch.device(device)
sam.to(device=torch_device)
predictor = SamPredictor(sam)
# calculate image embeddings
predictor.set_image(input_image)
```

Here is where I get this error:

```
File \"/app/repo/src/main.py\", line 150, in set_image_data
predictor.set_image(input_image)
File \"/usr/local/lib/python3.8/dist-packages/segment_anything/predictor.py\", line 60, in set_image
self.set_torch_image(input_image_torch, image.shape[:2])
File \"/usr/local/lib/python3.8/dist-packages/torch/autograd/grad_mode.py\", line 27, in decorate_context
return func(*args, **kwargs)
File \"/usr/local/lib/python3.8/dist-packages/segment_anything/predictor.py\", line 89, in set_torch_image
self.features = self.model.image_encoder(input_image)
File \"/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py\", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File \"/usr/local/lib/python3.8/dist-packages/segment_anything/modeling/image_encoder.py\
x = blk(x)
File \"/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py\", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File \"/usr/local/lib/python3.8/dist-packages/segment_anything/modeling/image_encoder.py\", line 174, in forward
x = self.attn(x)
File \"/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py\", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File \"/usr/local/lib/python3.8/dist-packages/segment_anything/modeling/image_encoder.py\", line 234, in forward
attn = add_decomposed_rel_pos(attn, q, self.rel_pos_h, self.rel_pos_w, (H, W), (H, W))
File \"/usr/local/lib/python3.8/dist-packages/segment_anything/modeling/image_encoder.py\", line 349, in add_decomposed_rel_pos
Rh = get_rel_pos(q_h, k_h, rel_pos_h)
File \"/usr/local/lib/python3.8/dist-packages/segment_anything/modeling/image_encoder.py\", line 322, in get_rel_pos
return rel_pos_resized[relative_coords.long()]
RuntimeError: CUDA error: an illegal memory access was encountered
```

I made some research on this issue and know that this issue can be caused by having not enough free GPU memory. I run nvidia-smi command and checked that I have enough GPU memory, but I am still facing this issue. My device is 12 GB Tesla K80.