huggingface / huggingface/setfit

setfit can not get good result for chinese language?

Open
#539 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.8k
Forks
267
Avg merge
36m
Merged PRs (30d)
5

Description

The number of data labels is 4. the eval loss increase and the train loss decrease. I think it is overfit. Code is as below:
# code
```from datasets import load_dataset
from setfit import SetFitModel, Trainer, TrainingArguments, sample_dataset, SetFitModelCardData
import ujson as json
from datasets import load_dataset, DatasetDict, Dataset
from transformers import EarlyStoppingCallback
import datetime

def convert_files_to_dataset(train_path, val_path):
# 定义一个函数来读取单个文件
def read_file(file_path):
with open(file_path, 'r', encoding='utf-8') as f:
lines = f.readlines()
data = [json.loads(line.strip()) for line in lines]
return data

# 读取训练集和验证集
train_data = read_file(train_path)
val_data = read_file(val_path)

# 将数据转换为Dataset
train_dataset = Dataset.from_list(train_data)
val_dataset = Dataset.from_list(val_data)

# 创建DatasetDict
dataset_dict = DatasetDict({'train': train_dataset, 'validation': val_dataset})

return dataset_dict

# 使用函数
train_path = 'train_cn.txt'
val_path = 'val_cn.txt'
dataset = convert_files_to_dataset(train_path, val_path)

from setfit import sample_dataset

train_dataset = sample_dataset(dataset["train"], num_samples=50)
print(train_dataset)

eval_dataset = dataset["validation"]
print(eval_dataset)
from setfit import SetFitModel

model = SetFitModel.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2', model_card_data=SetFitModelCardData(
language=['en', 'de', 'nl'],
))
from sentence_transformers.losses import CosineSimilarityLoss

from setfit import SetFitTrainer

trainer = SetFitTrainer(
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss_class=CosineSimilarityLoss,
num_iterations=20,
num_epochs=5
)
trainer.train()
metrics = trainer.evaluate()
print(metrics)```

log is as below:
```
Dataset({
features: ['text', 'label'],
num_rows: 200
})
Dataset({
features: ['text', 'label'],
num_rows: 40
})
/usr/local/matrix/conda3/envs/peft/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
model_head.pkl not found on HuggingFace Hub, initialising classification head with random weights. You should TRAIN this model on a downstream task to use it for predictions and inference.
b.py:51: DeprecationWarning: `SetFitTrainer` has been deprecated and will be removed in v2.0.0 of SetFit. Please use `Trainer` instead.
trainer = SetFitTrainer(
Using `evaluation_strategy="steps"` as `eval_steps` is defined.
Map: 100%|████████████████████████████████████████████████████████████████████████████| 200/200 [00:00<00:00, 12981.44 examples/s]
***** Running training *****
Num unique pairs = 8000
Batch size = 16
Num epochs = 5
Total optimization steps = 2500
0%| | 0/2500 [00:00

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the provided Python script with the Chinese train_cn.txt and val_cn.txt datasets, then inspect the SetFitTrainer configuration and logged embedding losses. Compare the deprecated SetFitTrainer usage with the current Trainer API and verify the multilingual model settings. Done means identifying a reproducible cause or clarifying that the result follows from the data and training configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.