[REQUEST] LLaMA-30B can not support 8K sequence length

I am reaching out to report an issue I encountered while implementing sequence parallelism with the LLaMA 30B model in DeepSpeed. The model configuration I used includes 48 attention heads. According to the literature and documentation, I expected to be able to use a sequence parallel dimension of 64. However, in practice, I found that the maximum feasible dimension I could set was only 16.

This limitation significantly impacts the performance of the Ulysses tool, particularly in handling sequence lengths. Even with sequences as short as 8K, Ulysses struggles to function effectively, which seems contrary to expectations given the capabilities of the LLaMA 30B model.

Could you please provide some insights or guidance on whether this is a known limitation or if there might be a potential configuration issue on my end? Any advice on how to achieve better performance with larger sequence parallel dimensions would be greatly appreciated.

Thank you for your time and assistance. @samadejacobs

Start by reproducing the reported LLaMA-30B configuration with 48 attention heads and compare sequence-parallel dimensions of 16 and 64 at an 8K sequence length. The issue names the Ulysses tool but no files, tests, or entry points; first locate its sequence-parallel configuration and establish whether the limitation is expected or a configuration failure. Done requires a documented explanation or a confirmed fix with validation.