Long sequence length (1M+ seq len) memory efficiency

We would like to be able to train models up to 1M seq len. The number of nodes necessary should also be reasonable. We are ok if the training runs take more time i.e. no runtime efficiency constraints, we just want to be able to do it.

The issue names no files, tests, or entry points. Start by locating the training path that controls sequence length and measuring memory and node requirements at large lengths. Done should demonstrate training at 1M+ sequence length with a reasonable node count, without requiring runtime efficiency.

Written by the indexing model from the issue text.