modelscope / modelscope/ms-swift
gme训练多batchsize使用自己设置的negative负样本报错,直接使用batch内采样正常
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Describe the bug
What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程,最好有截图)
gme训练多batchsize使用自己设置的negative负样本报错,直接使用batch内采样正常
Your hardware and system info
export MIN_PIXELS=200704 #2828256
#export MAX_PIXELS=1003520 #28281280
export SIZE_FACTOR=8
export MAX_PIXELS=602112
#MAX_PIXELS=245760
NNODES=$NNODES
NODE_RANK=$NODE_RANK
MASTER_ADDR=$MASTER_ADDR
MASTER_PORT=$MASTER_PORT
NPROC_PER_NODE=$NPROC_PER_NODE \
swift sft
--model
--train_type full
--freeze_aligner False
--dataset ''
--torch_dtype float16
--num_train_epochs 3
--per_device_train_batch_size 2
--gradient_accumulation_steps $(expr 64 / $NPROC_PER_NODE / $NNODES)
--save_steps 50
--logging_steps 5
--output_dir $DATA_OUTPUT_DIR
--lazy_tokenize true
--warmup_ratio 0.05
--learning_rate 5e-6
--deepspeed zero3
--dataloader_num_workers 4
--task_type embedding
--loss_type infonce
--dataloader_drop_last true
--logging_dir ${TENSORBOARD_LOG_DIR}
--gradient_checkpointing true
--report_to tensorboard
--system 'goods understanding.' $@ 2>&1 | tee -a $LOG_PATH
Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure through the swift sft command with --task_type embedding, --loss_type infonce, multiple batch sizes, and custom negative samples. Compare it with the in-batch sampling configuration and inspect the attached error output; done means custom negatives work without regressing the working sampling path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100