ByteDance-Seed / ByteDance-Seed/Bagel

The Size of ema.safetensors.

Open
#160 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
6.2k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

The size of the ema.safetensors saved during the training phase is nearly twice as large as the original official ema file.
official ema.safetensors: 27.21GB
my training ema.safetensors: 52.79GB
Causing out-of-memory (OOM) during reasoning.

Here is my scripts of training:
``` bash
torchrun \
--nnodes=1 \
--node_rank=0 \
--nproc_per_node=8 \
--master_addr=127.0.0.1 \
--master_port=12345 \
train/pretrain_unified_navit.py \
--dataset_config_file ./data/configs/t2i.yaml \
--layer_module Qwen2MoTDecoderLayer \
--max_latent_size 64 \
--model_path $model_path \
--use_flex True \
--resume-from $model_path \
--finetune_from_hf True \
--auto_resume True \
--resume-model-only True \
--finetune-from-ema True \
--wandb_offline True \
--results_dir $output_path \
--checkpoint_dir $ckpt_path \
--log_every 10 \
--save_every 1000 \
--lr 2e-5 \
--total_steps 10000 \
--visual_und False\
--expected_num_tokens 10240 \
--max_num_tokens 11520 \
--max_num_tokens_per_sample 10240 \
--num_workers
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in train/pretrain_unified_navit.py and trace how the training command creates or saves ema.safetensors, especially with --finetune-from-ema and checkpoint options. Compare the saved checkpoint contents and tensor sizes with the official 27.21GB file, then verify that the resulting checkpoint does not cause the reported out-of-memory failure during reasoning.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.