Execution exits unexpectedly: llava_llama3_8b_instruct_qlora_clip_vit_large_p14_336_e1_gpu1_finetune

2024/05/24 22:12:15 - mmengine - INFO -
------------------------------------------------------------
System environment:
sys.platform: linux
Python: 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 1024
GPU 0,1: NVIDIA GeForce RTX 4090
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.1, V12.1.66
GCC: x86_64-linux-gnu-gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0
PyTorch: 2.3.0+cu121
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 12.1
- NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
- CuDNN 8.9.2
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.3.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.18.0+cu121
OpenCV: 4.9.0
MMEngine: 0.10.4

Runtime environment:
launcher: none
randomness: {'seed': 1024, 'deterministic': False}
cudnn_benchmark: False
mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
dist_cfg: {'backend': 'nccl'}
seed: 1024
deterministic: False
Distributed launcher: none
Distributed training: False
GPU number: 1
------------------------------------------------------------

2024/05/24 22:12:15 - mmengine - INFO - Config:
SYSTEM = ''
accumulative_counts = 128
batch_size = 1
betas = (
0.9,
0.999,
)
custom_hooks = [
dict(
tokenizer=dict(
padding_side='right',
pretrained_model_name_or_path='./Meta-Llama-3-8B-Instruct',
trust_remote_code=True,
type='transformers.AutoTokenizer.from_pretrained'),
type='xtuner.engine.hooks.DatasetInfoHook'),
dict(
evaluation_images='https://llava-vl.github.io/static/images/view.jpg',
evaluation_inputs=[
'请描述一下这张照片',
'Please describe this picture',
],
every_n_iters=50000,
image_processor=dict(
pretrained_model_name_or_path='./clip-vit-large-patch14-336',
trust_remote_code=True,
type='transformers.CLIPImageProcessor.from_pretrained'),
prompt_template='xtuner.utils.PROMPT_TEMPLATE.llama3_chat',
system='',
tokenizer=dict(
padding_side='right',
pretrained_model_name_or_path='./Meta-Llama-3-8B-Instruct',
trust_remote_code=True,
type='transformers.AutoTokenizer.from_pretrained'),
type='xtuner.engine.hooks.EvaluateChatHook'),
]
data_path = './data/llava_data/LLaVA-Instruct-150K/llava_v1_5_mix665k.json'
data_root = './data/llava_data/'
dataloader_num_workers = 0
default_hooks = dict(
checkpoint=dict(
by_epoch=False,
interval=50000,
max_keep_ckpts=2,
type='mmengine.hooks.CheckpointHook'),
logger=dict(
interval=10,
log_metric_by_epoch=False,
type='mmengine.hooks.LoggerHook'),
param_scheduler=dict(type='mmengine.hooks.ParamSchedulerHook'),
sampler_seed=dict(type='mmengine.hooks.DistSamplerSeedHook'),
timer=dict(type='mmengine.hooks.IterTimerHook'))
env_cfg = dict(
cudnn_benchmark=False,
dist_cfg=dict(backend='nccl'),
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
evaluation_freq = 50000
evaluation_images = 'https://llava-vl.github.io/static/images/view.jpg'
evaluation_inputs = [
'请描述一下这张照片',
'Please describe this picture',
]
image_folder = './data/llava_data/llava_images'
image_processor = dict(
pretrained_model_name_or_path='./clip-vit-large-patch14-336',
trust_remote_code=True,
type='transformers.CLIPImageProcessor.from_pretrained')
launcher = 'none'
llava_dataset = dict(
data_path='./data/llava_data/LLaVA-Instruct-150K/llava_v1_5_mix665k.json',
dataset_map_fn='xtuner.dataset.map_fns.llava_map_fn',
image_folder='./data/llava_data/llava_images',
image_processor=dict(
pretrained_model_name_or_path='./clip-vit-large-patch14-336',
trust_remote_code=True,
type='transformers.CLIPImageProcessor.from_pretrained'),
max_length=1472,
pad_image_to_square=True,
template_map_fn=dict(
template='xtuner.utils.PROMPT_TEMPLATE.llama3_chat',
type='xtuner.dataset.map_fns.template_map_fn_factory'),
tokenizer=dict(
padding_side='right',
pretrained_model_name_or_path='./Meta-Llama-3-8B-Instruct',
trust_remote_code=True,
type='transformers.AutoTokenizer.from_pretrained'),
type='xtuner.dataset.LLaVADataset')
llm_name_or_path = './Meta-Llama-3-8B-Instruct'
load_from = None
log_level = 'INFO'
log_processor = dict(by_epoch=False)
lr = 0.0002
max_epochs = 1
max_length = 1472
max_norm = 1
model = dict(
freeze_llm=True,
freeze_visual_encoder=True,
llm=dict(
pretrained_model_name_or_path='./Meta-Llama-3-8B-Instruct',
quantization_config=dict(
bnb_4bit_compute_dtype='torch.float16',
bnb_4bit_quant_type='nf4',
bnb_4bit_use_double_quant=True,
llm_int8_has_fp16_weight=False,
llm_int8_threshold=6.0,
load_in_4bit=True,
load_in_8bit=False,
type='transformers.BitsAndBytesConfig'),
torch_dtype='torch.float16',
trust_remote_code=True,
type='transformers.AutoModelForCausalLM.from_pretrained'),
llm_lora=dict(
bias='none',
lora_alpha=16,
lora_dropout=0.05,
r=64,
task_type='CAUSAL_LM',
type='peft.LoraConfig'),
pretrained_pth=
'./work_dirs/llava_llama3_8b_instruct_quant_clip_vit_large_p14_336_e1_gpu1_pretrain/iter_558128.pth',
type='xtuner.model.LLaVAModel',
visual_encoder=dict(
pretrained_model_name_or_path='./clip-vit-large-patch14-336',
type='transformers.CLIPVisionModel.from_pretrained'))
optim_type = 'torch.optim.AdamW'
optim_wrapper = dict(
optimizer=dict(
betas=(
0.9,
0.999,
),
lr=0.0002,
type='torch.optim.AdamW',
weight_decay=0),
type='DeepSpeedOptimWrapper')
param_scheduler = [
dict(
begin=0,
by_epoch=True,
convert_to_iter_based=True,
end=0.03,
start_factor=1e-05,
type='mmengine.optim.LinearLR'),
dict(
begin=0.03,
by_epoch=True,
convert_to_iter_based=True,
end=1,
eta_min=0.0,
type='mmengine.optim.CosineAnnealingLR'),
]
pretrained_pth = './work_dirs/llava_llama3_8b_instruct_quant_clip_vit_large_p14_336_e1_gpu1_pretrain/iter_558128.pth'
prompt_template = 'xtuner.utils.PROMPT_TEMPLATE.llama3_chat'
randomness = dict(deterministic=False, seed=1024)
resume = False
runner_type = 'FlexibleRunner'
save_steps = 50000
save_total_limit = 2
strategy = dict(
config=dict(
bf16=dict(enabled=True),
fp16=dict(enabled=False, initial_scale_power=16),
gradient_accumulation_steps='auto',
gradient_clipping='auto',
train_micro_batch_size_per_gpu='auto',
zero_allow_untested_optimizer=True,
zero_force_ds_cpu_optimizer=False,
zero_optimization=dict(overlap_comm=True, stage=2)),
exclude_frozen_parameters=True,
gradient_accumulation_steps=128,
gradient_clipping=1,
train_micro_batch_size_per_gpu=1,
type='xtuner.engine.DeepSpeedStrategy')
tokenizer = dict(
padding_side='right',
pretrained_model_name_or_path='./Meta-Llama-3-8B-Instruct',
trust_remote_code=True,
type='transformers.AutoTokenizer.from_pretrained')
train_cfg = dict(max_epochs=1, type='xtuner.engine.runner.TrainLoop')
train_dataloader = dict(
batch_size=1,
collate_fn=dict(type='xtuner.dataset.collate_fns.default_collate_fn'),
dataset=dict(
data_path=
'./data/llava_data/LLaVA-Instruct-150K/llava_v1_5_mix665k.json',
dataset_map_fn='xtuner.dataset.map_fns.llava_map_fn',
image_folder='./data/llava_data/llava_images',
image_processor=dict(
pretrained_model_name_or_path='./clip-vit-large-patch14-336',
trust_remote_code=True,
type='transformers.CLIPImageProcessor.from_pretrained'),
max_length=1472,
pad_image_to_square=True,
template_map_fn=dict(
template='xtuner.utils.PROMPT_TEMPLATE.llama3_chat',
type='xtuner.dataset.map_fns.template_map_fn_factory'),
tokenizer=dict(
padding_side='right',
pretrained_model_name_or_path='./Meta-Llama-3-8B-Instruct',
trust_remote_code=True,
type='transformers.AutoTokenizer.from_pretrained'),
type='xtuner.dataset.LLaVADataset'),
num_workers=0,
sampler=dict(
length_property='modality_length',
per_device_batch_size=128,
type='xtuner.dataset.samplers.LengthGroupedSampler'))
visual_encoder_name_or_path = './clip-vit-large-patch14-336'
visualizer = None
warmup_ratio = 0.03
weight_decay = 0
work_dir = './work_dirs/llava_llama3_8b_instruct_qlora_clip_vit_large_p14_336_e1_gpu1_finetune'

2024/05/24 22:12:16 - mmengine - WARNING - Failed to search registry with scope "mmengine" in the "builder" registry tree. As a workaround, the current "builder" registry in "xtuner" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmengine" is a correct scope, or whether the registry is initialized.