help in reproducing the WikiHop results using the entire sequence length?

Hi,

I am trying to reproduce results for some downstream tasks in order to better understand Longformer. Paper - Page 12 - explains how these tasks are implemented but I couldn't find any code in the repo.


Any help with some task-specific code snippet?

Also, how does the forward layer work for each of the chunks passed? For instance, say there is a document with 5x 4096 (Longformer-base) chunks and a label **1** (tag). When I pass such 5 chunks in the forward pass, it will simply create output activations without any gradient calculation. Then all activations are **concatenated** into a long sequence. So if there two documents with 5 and 9 chunks respectively, then each will be [5x 768] and [9x 768], different shapes? So just mean/max pool after stacking them? When does the backward pass happen if I want to use my custom **classifier** instead of AutoModelForSequenceClassification?

Some more elaborate explanation would be helpful.

Thanks!