Memory-efficient attention (without xformers)

I implemented sub-quadratic attention (as described in https://arxiv.org/abs/2112.05682v2):
https://twitter.com/Birchlabs/status/1607503573906063362
https://github.com/Birch-san/diffusers/pull/1
https://github.com/Birch-san/diffusers-play/commit/a573e3d9ea4fdacfdee7ddd5eecdac29b236fc00

is this worth upstreaming? it enables creation of images larger than can be achieved with attention slicing.

Start by reading the linked paper and the referenced implementation and commit, then compare their sub-quadratic attention approach with the repository's current attention path. Done means determining an upstreamable implementation that works without xformers and enables image sizes beyond those possible with attention slicing, with appropriate validation.

Vom Indexierungsmodell aus dem Issue-Text verfasst.