Hash embeddings abusing

I think you're taking advantage of the fact that embeddings are essentially 0-flop lookup tables. Your hash embeddings have 6 * 500k * 1024 = 3 billion parameters, which adds additional capacity to the model, but in terms of raw FLOPs, it contributes nothing. So, I think it would be more fair to compare this setup with a BPE model that also has a similarly large embedding matrix