google-research / google-research/language

ORQA/REALM - Invalid argument: indices[0] = 10311577 is not in

Open
#89 15 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.8k
Forks
362
PR merge metrics
No merged PRs in 30d

Description

I am trying to run Orqa finetuning with Realm models on my own data set. I have created the blocks.tfr file and used --num_block_records=154800 which is the number of rows in my records file.

I launch fine-tuning with

`python -m language.orqa.experiments.orqa_experiment --retriever_module_path=$(pwd)/language/orqa/gs/realm-data/cc_news_pretrained/embedder --reader_module_path=$(pwd)/language/orqa/gs/realm-data/cc_news_pretrained/bert --block_records_path=$(pwd)/language/orqa/gs/orqa-data/trump-blocks/blocks.tfr --num_block_records=227246 --data_root=$(pwd)/language/orqa/gs/orqa-data/resplit --model_dir=$(pwd)/language/orqa/models/orqa_trump_model_from_realm --dataset_name=TrumpQuestions --num_train_steps=$(( 3417 * 20 )) --save_checkpoints_steps=1000`

Everything seems to go well in the first ten minutes, but once shuffle buffer filling completes, I get an error

```
2020-10-13 13:06:30.130961: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:199] Shuffle buffer filled.
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/orqa/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1367, in _do_call
return fn(*args)
File "/home/ubuntu/anaconda3/envs/orqa/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1352, in _run_fn
target_list, run_metadata)
File "/home/ubuntu/anaconda3/envs/orqa/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1445, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[0] = 10311577 is not in [0, 227246)
[[{{node GatherV2_2}}]]
(1) Invalid argument: indices[0] = 10311577 is not in [0, 227246)
[[{{node GatherV2_2}}]]
[[reader/RaggedRange_4/_8285]]
0 successful operations.
0 derived errors ignored.
....
....
Errors may have originated from an input operation.
Input Source operations connected to node GatherV2_2:
blocks/read (defined at /ioannis/language/language/orqa/models/orqa_model.py:147)
Squeeze (defined at /ioannis/language/language/orqa/models/orqa_model.py:129)

Input Source operations connected to node GatherV2_2:
blocks/read (defined at /ioannis/language/language/orqa/models/orqa_model.py:147)
Squeeze (defined at /ioannis/language/language/orqa/models/orqa_model.py:129)

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.