[prefixLM] Investigate cuda kernels

Up until recently we've been using pytorch code in order to apply "scale -> mask -> softmax" in attention mechanism for prefix LM.

I've recently discovered that there exists two cuda kernels to attention matrices:
- `ScaledUpperTriangMaskedSoftmax` which is as the naming suggest specific to GPT style models
- `ScaledMaskedSoftmax`

The later one hasn't really been tested, and since I didn't have time to deepdive in the cuda code, I decided not to use it for initial prefix lm. However https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/151 has removed the mechanism to force prefix lm to use the pytorch route.

In this issue, we want to:
- [ ] make sure that `ScaledMaskedSoftmax` support prefix attention mask if fed with one.
- [ ] if it doesn't, implement a prefix lm cuda kernel.

@RezaYazdaniAminabadi , If you could provide us with your expertise on that one, maybe just checking the first item that'd be great. Thank you!

Review the two named kernels, `ScaledMaskedSoftmax` and `ScaledUpperTriangMaskedSoftmax`, along with the prefix-LM change discussed in pull request #151. First determine whether `ScaledMaskedSoftmax` accepts prefix attention masks; done means confirming that behavior or implementing a prefix-LM CUDA kernel if it does not.

Written by the indexing model from the issue text.