Can we make transformers work here?

Transformer architectures have revealed themselves to be the unquestionable state-of-the-art when it comes to sequence-to-sequence modelling architectures, and would be incredibly valuable to implement here. The issue in our use case is that the outer product in the self-attention mechanism makes the time and memory quadratic in the length of the timeseries. With our current sampling rate of 4096, that makes even a few seconds a very like timeseries in terms of number of steps (though in the context of #23 and #26 it's possible we don't need as many second as we're using now). However, the blog post linked to above outlines a couple a schemes in the literature for addressing this limitation. It's worth looking into what we might be able to adopt and seeing how that can improve things.

Start by reading the linked transformer architectures article and the context in issues #23 and #26. Investigate approaches that avoid quadratic time and memory at the current sampling rate, then define an implementation approach and demonstrate improved sequence-to-sequence performance on the project's timeseries use case.