huggingface / huggingface/setfit
setfit can not get good result for chinese language?
- Dominant language
- Jupyter Notebook
- Stars
- 2.8k
- Forks
- 267
- Avg merge
- 36m
- Merged PRs (30d)
- 5
Description
The number of data labels is 4. the eval loss increase and the train loss decrease. I think it is overfit. Code is as below:
# code
```from datasets import load_dataset
from setfit import SetFitModel, Trainer, TrainingArguments, sample_dataset, SetFitModelCardData
import ujson as json
from datasets import load_dataset, DatasetDict, Dataset
from transformers import EarlyStoppingCallback
import datetime
def convert_files_to_dataset(train_path, val_path):
# 定义一个函数来读取单个文件
def read_file(file_path):
with open(file_path, 'r', encoding='utf-8') as f:
lines = f.readlines()
data = [json.loads(line.strip()) for line in lines]
return data
# 读取训练集和验证集
train_data = read_file(train_path)
val_data = read_file(val_path)
# 将数据转换为Dataset
train_dataset = Dataset.from_list(train_data)
val_dataset = Dataset.from_list(val_data)
# 创建DatasetDict
dataset_dict = DatasetDict({'train': train_dataset, 'validation': val_dataset})
return dataset_dict
# 使用函数
train_path = 'train_cn.txt'
val_path = 'val_cn.txt'
dataset = convert_files_to_dataset(train_path, val_path)
from setfit import sample_dataset
train_dataset = sample_dataset(dataset["train"], num_samples=50)
print(train_dataset)
eval_dataset = dataset["validation"]
print(eval_dataset)
from setfit import SetFitModel
model = SetFitModel.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2', model_card_data=SetFitModelCardData(
language=['en', 'de', 'nl'],
))
from sentence_transformers.losses import CosineSimilarityLoss
from setfit import SetFitTrainer
trainer = SetFitTrainer(
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss_class=CosineSimilarityLoss,
num_iterations=20,
num_epochs=5
)
trainer.train()
metrics = trainer.evaluate()
print(metrics)```
log is as below:
```
Dataset({
features: ['text', 'label'],
num_rows: 200
})
Dataset({
features: ['text', 'label'],
num_rows: 40
})
/usr/local/matrix/conda3/envs/peft/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
model_head.pkl not found on HuggingFace Hub, initialising classification head with random weights. You should TRAIN this model on a downstream task to use it for predictions and inference.
b.py:51: DeprecationWarning: `SetFitTrainer` has been deprecated and will be removed in v2.0.0 of SetFit. Please use `Trainer` instead.
trainer = SetFitTrainer(
Using `evaluation_strategy="steps"` as `eval_steps` is defined.
Map: 100%|████████████████████████████████████████████████████████████████████████████| 200/200 [00:00<00:00, 12981.44 examples/s]
***** Running training *****
Num unique pairs = 8000
Batch size = 16
Num epochs = 5
Total optimization steps = 2500
0%| | 0/2500 [00:00
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the provided Python script with the Chinese train_cn.txt and val_cn.txt datasets, then inspect the SetFitTrainer configuration and logged embedding losses. Compare the deprecated SetFitTrainer usage with the current Trainer API and verify the multilingual model settings. Done means identifying a reproducible cause or clarifying that the result follows from the data and training configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100