The number of tokens in a batch may exceeds max_batch_size when batch_type is "tokens".

Hello!

Using the score_batch feature with batch_type=tokens, I found that the get_batch_ size_increment function increments the batch size by the length of each example, rather than the longest example in the given sentences.
This may produce a batch with more tokens than max_batch_size. The actual size may be max_batch_size + the number of padding tokens.

https://github.com/OpenNMT/CTranslate2/blob/0455e1fe1e4a57e00d13d1b13ad44a61545ccfe9/src/batch_reader.cc#L22

As a small experiment, I compared the input file sorted by sentence length in reverse order with the unsorted original. And I found that the process ended successfully with the one sorted, even though the original unsorted one produced an out-of-memory error using the same max_batch_size. I think it happened because the actual number of tokens in the batch was so large because the unsorted input file had more padding tokens.

Start in src/batch_reader.cc at the linked get_batch_size_increment function and trace how score_batch handles batch_type="tokens". Reproduce the issue with differently ordered inputs and the same max_batch_size. Done means token batches stay within max_batch_size and the reported out-of-memory behavior no longer occurs.