google-research / google-research/language
ORQA/REALM - Invalid argument: indices[0] = 10311577 is not in
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 362
- PR merge metrics
- No merged PRs in 30d
Description
I am trying to run Orqa finetuning with Realm models on my own data set. I have created the blocks.tfr file and used --num_block_records=154800 which is the number of rows in my records file.
I launch fine-tuning with
`python -m language.orqa.experiments.orqa_experiment --retriever_module_path=$(pwd)/language/orqa/gs/realm-data/cc_news_pretrained/embedder --reader_module_path=$(pwd)/language/orqa/gs/realm-data/cc_news_pretrained/bert --block_records_path=$(pwd)/language/orqa/gs/orqa-data/trump-blocks/blocks.tfr --num_block_records=227246 --data_root=$(pwd)/language/orqa/gs/orqa-data/resplit --model_dir=$(pwd)/language/orqa/models/orqa_trump_model_from_realm --dataset_name=TrumpQuestions --num_train_steps=$(( 3417 * 20 )) --save_checkpoints_steps=1000`
Everything seems to go well in the first ten minutes, but once shuffle buffer filling completes, I get an error
```
2020-10-13 13:06:30.130961: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:199] Shuffle buffer filled.
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/orqa/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1367, in _do_call
return fn(*args)
File "/home/ubuntu/anaconda3/envs/orqa/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1352, in _run_fn
target_list, run_metadata)
File "/home/ubuntu/anaconda3/envs/orqa/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1445, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[0] = 10311577 is not in [0, 227246)
[[{{node GatherV2_2}}]]
(1) Invalid argument: indices[0] = 10311577 is not in [0, 227246)
[[{{node GatherV2_2}}]]
[[reader/RaggedRange_4/_8285]]
0 successful operations.
0 derived errors ignored.
....
....
Errors may have originated from an input operation.
Input Source operations connected to node GatherV2_2:
blocks/read (defined at /ioannis/language/language/orqa/models/orqa_model.py:147)
Squeeze (defined at /ioannis/language/language/orqa/models/orqa_model.py:129)
Input Source operations connected to node GatherV2_2:
blocks/read (defined at /ioannis/language/language/orqa/models/orqa_model.py:147)
Squeeze (defined at /ioannis/language/language/orqa/models/orqa_model.py:129)
```
Contributor guide
Assessment
This issue has not been assessed yet.