Truncation performs slowly. Tokenizer firstly encodes long sequence and then truncates it.

I work with very long sequences, which I want to truncate to a shorter one in the pre-processing stage.
I found that the Tokenizer first encodes the whole sequence whole, after that truncates it. Is there a way to truncate it on the fly, to save time on tokenization of unused part of the sequence?

I use transformers 4.42.4.
tokenizer class is LlamaTokenizerFast

The code and the output:

```
from transformers import AutoTokenizer
from time import time

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/deepseek-coder-1.3b-base")
# tokenizer class is LlamaTokenizerFast
def encode_long(meta_len, batch_size=200):
text = 500*"jibber-jabber"
text_batch = batch_size*[meta_len*text]

start_time = time()
batch = tokenizer.batch_encode_plus(text_batch, truncation=True, max_length=1000)
print(f"Seq encoded len = {len(batch['input_ids'][0])}; Time used = {time() - start_time:.2f}, Symbols = {meta_len*len(text)}")

encode_long(meta_len=1)
encode_long(meta_len=10)
encode_long(meta_len=100)
```

Results of the code on my machine:

Seq encoded len = 1000; Time used = 0.07, Symbols = 6500
Seq encoded len = 1000; Time used = 0.59, Symbols = 65000
Seq encoded len = 1000; Time used = 8.98, Symbols = 650000

[Colab ](https://colab.research.google.com/drive/1HkEBto8wgL4MbBp4dqZJ0hH_6KNUS1Ka?usp=sharing) for reproduction.