Code to generate longformer-base-4096

Hello,
I wonder if you have the code that generated [this pre-trained model](https://huggingface.co/allenai/longformer-base-4096#).
(or [longformer-large-4096-finetuned-triviaqa](https://huggingface.co/allenai/longformer-large-4096-finetuned-triviaqa))

I followed the steps in [your notebook](https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb) and updated the[ triviaqa.py](https://github.com/allenai/longformer/blob/40377ddc43d5d446385503dd8255f7e75ded7811/scripts/triviaqa.py) file; it is attached.

However, I get some miss match dimension in linear operations as follow:
Traceback (most recent call last):
File "/home/arashari/longformer_original/scripts/triviaqa.py", line 760, in
main(args)
File "/home/arashari/longformer_original/scripts/triviaqa.py", line 752, in main
trainer.fit(model)
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 979, in fit
self.single_gpu_train(model)
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/distrib_parts.py", line 185, in single_gpu_train
self.run_pretrain_routine(model)
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 1139, in run_pretrain_routine
False)
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 293, in _evaluate
output = self.evaluation_forward(model, batch, batch_idx, dataloader_idx, test_mode)
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 485, in evaluation_forward
output = model.validation_step(*args)
File "/home/arashari/longformer_original/scripts/triviaqa.py", line 440, in validation_step
output = self.forward(input_ids, input_mask, segment_ids, subword_starts, subword_ends)
File "/usr/local/lib/python3.6/dist-packages/apex/amp/_initialize.py", line 197, in new_fwd
**applier(kwargs, input_caster))
File "/home/arashari/longformer_original/scripts/triviaqa.py", line 362, in forward
logits = self.qa_outputs(sequence_output)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1610, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch, m1: [1 x 2376], m2: [768 x 2] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:283

Run time arguments are:
python -m triviaqa \
--train_dataset output/squad-wikipedia-train-4096.json \
--dev_dataset squad-wikipedia-dev-4096.json \
--gpus 0,1,2,3,4,5,6,7 --batch_size 1 --num_workers 4 \
--lr 0.00003 --warmup 1000 --epochs 4 --max_seq_len 4096 --doc_stride -1 \
--save_prefix data/output/ \
--seed 4321 \
--attention_mode 'sliding_chunks' \
--model_path roberta-base

Any comment/suggestion on this?

Further, I tried the [run_squad](https://github.com/huggingface/transformers/tree/master/examples/question-answering) from huggingface and seems like the converter (triviaqa to squad) does not add title; it is not compatible with this example.
For example it is complaining about fields such as "title".
Are you aware of any other converter?
[triviaqa.txt](https://github.com/allenai/longformer/files/4887535/triviaqa.txt)