google-research / google-research/language

Memory requirements to train bert-joint question answering model?

Open
#29 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.8k
Forks
362
PR merge metrics
No merged PRs in 30d

Description

Hi,

To replicate the results, I am trying to train the bert-joint model using the below command:

`python -m language.question_answering.bert_joint.run_nq \
--logtostderr \
--bert_config_file=wwm_uncased_L-24_H-1024_A-16/bert_config.json \
--vocab_file=wwm_uncased_L-24_H-1024_A-16/vocab.txt \
--train_precomputed_file=bert-joint-baseline/nq-train.tfrecords-00000-of-00001 \
--train_num_precomputed=494670 \
--learning_rate=3e-5 \
--num_train_epochs=1 \
--max_seq_length=512 \
--save_checkpoints_steps=5000 \
--init_checkpoint=uncased_L-24_H-1024_A-16/bert_model.ckpt \
--do_train \
--output_dir=baseline \
--use_tpu=False \
--output_prediction_file=baseline/predictions.json`

But I am running into Out of Memory issues.
What are the minimum requirements to train the model. I am using GPU for training.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.