[Bug]: tokenizer_config max_length caps several models at 128 tokens

What happened?

load_tokenizer takes min(model_max_length, max_length) when both keys exist in tokenizer_config.json (fastembed/common/preprocessor_utils.py:47). Several Qdrant-hosted ONNX repos carry a "max_length": 128 that the upstream model repo does not, so input is truncated at 128 tokens with no warning.

For two models this contradicts the shipped description:

Two more repos carry the same key where the upstream repo has none: Qdrant/Splade_PP_en_v1 (upstream sentence_bert_config.json is max_seq_length: 512) and Qdrant/all_miniLM_L6_v2_with_attentions (BM42). Their descriptions carry no token count.

qdrant/paraphrase-multilingual-MiniLM-L12-v2-onnx-Q also caps at 128, and there it matches the upstream. Whether the four above are deliberate is a question for maintainers.

What is the expected behaviour?

The effective limit matches the description, and a limit below the upstream default is a deliberate choice.

Related: there is no way to change the limit through the API. enable_truncation is called once from the config file, and no max_length kwarg reaches the tokenizer, so the only override today is specific_model_path with an edited tokenizer_config.json.

A minimal reproducible example

from fastembed import TextEmbedding, SparseTextEmbedding

print(TextEmbedding("sentence-transformers/all-MiniLM-L6-v2").model.tokenizer.truncation["max_length"])  # 128, description says 256

m = SparseTextEmbedding("prithivida/Splade_PP_en_v1")
doc = "lorem ipsum " * 200 + "zebra"
print(m.model.tokenizer.token_to_id("zebra") in list(m.embed([doc]))[0].indices)  # False

The second check shows the effect: a term past position 128 is absent from the sparse vector.

What Python version are you on? e.g. python --version

Python 3.11, uv

FastEmbed version

0.8.0

What os are you seeing the problem on?

MacOS

Start with load_tokenizer in fastembed/common/preprocessor_utils.py:47 and run the minimal reproducer against the affected models. Compare tokenizer_config.json limits with the model descriptions and upstream settings, then determine how deliberate lower limits should be represented. Done means the effective limits match documented behavior and intentional caps are clear, including the API override question.