Question about the implemented sparse attention

Dear authors,

I have a question about sparse attention that you implemented. Does Longformer-loop have the same accuracy as Longformer-chunk? Also, slinding_window_no_overlap version that you implemented in the latest code has the same accuracy as Longformer-loop?

In your latest version of the paper, you commented Longformer-loop is a naive implementation that computes each diagonal separately in a loop. As I know, each token only computes with related window size tokens. This method is memory efficient, but it is inefficient on GPU/TPU as you said. Because of many gather and scatter operations.

So you implemented longformer-chunk that is efficient on GPU/TPU. However, Longformer-chunk method computes attention between chunked tokens. It is different from Longformer-loop. Some tokens compute attention operations with more than window size tokens because of blocking(chunk).

In the following picture, you can see the differences easily.

Grey is longformer-loop. Green, red, yellow and blue are longformer-chunk.
If I misunderstood your paper, please correct me. And, I'd appreciate you could help me.

Thanks!