ByteDance-Seed / ByteDance-Seed/Bagel
The Size of ema.safetensors.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.2k
- Forks
- 545
- PR merge metrics
- No merged PRs in 30d
Description
The size of the ema.safetensors saved during the training phase is nearly twice as large as the original official ema file.
official ema.safetensors: 27.21GB
my training ema.safetensors: 52.79GB
Causing out-of-memory (OOM) during reasoning.
Here is my scripts of training:
``` bash
torchrun \
--nnodes=1 \
--node_rank=0 \
--nproc_per_node=8 \
--master_addr=127.0.0.1 \
--master_port=12345 \
train/pretrain_unified_navit.py \
--dataset_config_file ./data/configs/t2i.yaml \
--layer_module Qwen2MoTDecoderLayer \
--max_latent_size 64 \
--model_path $model_path \
--use_flex True \
--resume-from $model_path \
--finetune_from_hf True \
--auto_resume True \
--resume-model-only True \
--finetune-from-ema True \
--wandb_offline True \
--results_dir $output_path \
--checkpoint_dir $ckpt_path \
--log_every 10 \
--save_every 1000 \
--lr 2e-5 \
--total_steps 10000 \
--visual_und False\
--expected_num_tokens 10240 \
--max_num_tokens 11520 \
--max_num_tokens_per_sample 10240 \
--num_workers
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in train/pretrain_unified_navit.py and trace how the training command creates or saves ema.safetensors, especially with --finetune-from-ema and checkpoint options. Compare the saved checkpoint contents and tensor sizes with the official 27.21GB file, then verify that the resulting checkpoint does not cause the reported out-of-memory failure during reasoning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100