huggingface / huggingface/setfit

Trainer crashes on GCP

Open
#264 5 comments 2 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.8k
Forks
267
Avg merge
36m
Merged PRs (30d)
5

Description

While running the same setup as given the [Text classification tutorial](https://github.com/huggingface/setfit/blob/main/notebooks/text-classification.ipynb), my notebook on Google Cloud Platform crashes without any further notice or error while calling the `trainer.train()`.

I tried to watch `nvidia-smi` during crash time but kernel crashes too quickly to notice anything. The machine definitely should be big enough. The specs are:

> 60 GB memory
1x P100 GPU
nvidia-drivers are installed
`torch.cuda.is_available()` returns `True`

The same code does run on Google Colab with no issues.

What can I try to debug? Thanks!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.