huggingface / huggingface/setfit

Kernel crash due to out of memory for large dataset

Open
#472 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.8k
Forks
267
Avg merge
36m
Merged PRs (30d)
5

Description

Hi team, I am using setfit for a multiclass classification problem (130+ classes). I have ~800,000K labelled samples as training set and ~200,000K as test set. I see my kernel crashing even though I have 1 batch on my 32GB RAM on my MacBook M1 Pro.

My Train and Test CSV has the same labels.

Code to reproduce:
```
import pandas as pd
from datasets import load_dataset

from setfit import SetFitModel, Trainer, TrainingArguments, sample_dataset

df = pd.read_csv("Combined Train.csv", usecols=['label'])
model = SetFitModel.from_pretrained("BAAI/bge-small-en-v1.5", labels=df.label)

dataset = load_dataset('csv', data_files={
"train": 'Combined Train.csv',
"test": 'Combined Test.csv'
})

# Preparing the training arguments
args = TrainingArguments(
batch_size=2,
num_epochs=1,
)

# Preparing the trainer
trainer = Trainer(
model=model,
args=args,
train_dataset=dataset['train']
)

trainer.train()
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.