Mistake in FLOPs calculations in the paper

Local models forwards at each byte, but debedder in BPE forwards at each token which is 4.4 times smaller in FLOPS. Considering your architecture hyper-parameters it seems like you missed that. In small models it gives advantage in terms of FLOPs for BLT for ±17%.

My calculations with proofs:
1) With your formulas: https://www.desmos.com/calculator/ippsrya4kf?lang=ru
2) With another formulas: https://colab.research.google.com/drive/1C3As0sG-SL482KslP-DxylpAYut-9nyM?usp=sharing #

Compare the paper's FLOPs formulas with the calculations in the linked Desmos and Colab documents, focusing on the local-model byte forwards and BPE token forwards described in the issue. Reproduce the discrepancy using the stated architecture hyperparameters; the work is done when the calculation is verified and the paper or its formulas are corrected or the concern is explained.

Written by the indexing model from the issue text.