huggingface / huggingface/setfit
Kernel crash due to out of memory for large dataset
- Dominant language
- Jupyter Notebook
- Stars
- 2.8k
- Forks
- 267
- Avg merge
- 36m
- Merged PRs (30d)
- 5
Description
Hi team, I am using setfit for a multiclass classification problem (130+ classes). I have ~800,000K labelled samples as training set and ~200,000K as test set. I see my kernel crashing even though I have 1 batch on my 32GB RAM on my MacBook M1 Pro.
My Train and Test CSV has the same labels.
Code to reproduce:
```
import pandas as pd
from datasets import load_dataset
from setfit import SetFitModel, Trainer, TrainingArguments, sample_dataset
df = pd.read_csv("Combined Train.csv", usecols=['label'])
model = SetFitModel.from_pretrained("BAAI/bge-small-en-v1.5", labels=df.label)
dataset = load_dataset('csv', data_files={
"train": 'Combined Train.csv',
"test": 'Combined Test.csv'
})
# Preparing the training arguments
args = TrainingArguments(
batch_size=2,
num_epochs=1,
)
# Preparing the trainer
trainer = Trainer(
model=model,
args=args,
train_dataset=dataset['train']
)
trainer.train()
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.