modelscope / modelscope/ms-swift

gme训练多batchsize使用自己设置的negative负样本报错,直接使用batch内采样正常

Open
#7,202 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug stale
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Describe the bug
What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程,最好有截图)
gme训练多batchsize使用自己设置的negative负样本报错,直接使用batch内采样正常

Your hardware and system info

export MIN_PIXELS=200704 #2828256
#export MAX_PIXELS=1003520 #28281280
export SIZE_FACTOR=8
export MAX_PIXELS=602112
#MAX_PIXELS=245760
NNODES=$NNODES
NODE_RANK=$NODE_RANK
MASTER_ADDR=$MASTER_ADDR
MASTER_PORT=$MASTER_PORT
NPROC_PER_NODE=$NPROC_PER_NODE \

swift sft
--model
--train_type full
--freeze_aligner False
--dataset '
'
--torch_dtype float16
--num_train_epochs 3
--per_device_train_batch_size 2
--gradient_accumulation_steps $(expr 64 / $NPROC_PER_NODE / $NNODES)
--save_steps 50
--logging_steps 5
--output_dir $DATA_OUTPUT_DIR
--lazy_tokenize true
--warmup_ratio 0.05
--learning_rate 5e-6
--deepspeed zero3
--dataloader_num_workers 4
--task_type embedding
--loss_type infonce
--dataloader_drop_last true
--logging_dir ${TENSORBOARD_LOG_DIR}
--gradient_checkpointing true
--report_to tensorboard
--system 'goods understanding.' $@ 2>&1 | tee -a $LOG_PATH

Additional context

Image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure through the swift sft command with --task_type embedding, --loss_type infonce, multiple batch sizes, and custom negative samples. Compare it with the in-batch sampling configuration and inspect the attached error output; done means custom negatives work without regressing the working sampling path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.