[Bug]: Docstring does not align with behaviour
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 248
- Avg merge
- 4d 8h
- Merged PRs (30d)
- 4
Description
What happened?
I was using the sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 model to embed chunks of text, and I noticed that while the docstring in the model list specifies 512 input tokens truncation the actual number of tokens before truncation is 128 in the tokenizer at embedder.tokenizer.
We tested the embedding procedure and it seems that the embedder is indeed using 128 long truncation, as is specified in the paraphrase documentation.
What is the expected behaviour?
The docstring and documentation should allign with actual token truncation behaviour.
A minimal reproducible example
import TextEmbedding
embedder = TextEmbedding(model_name="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
print(embedder.model.tokenizer.truncation) // Shows the actual token truncation window
// Truncation can be changed with
embedder.model.tokenizer.enable_truncation(max_size=512, padding=0)
// However this is not the default setting for paraphrase-multilingual
What Python version are you on? e.g. python --version
python3.12
FastEmbed version
v0.6.0
What os are you seeing the problem on?
Linux
Relevant stack traces and/or logs
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the model-list docstring for sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 and compare its stated truncation limit with the default at embedder.model.tokenizer. Update the docstring and related documentation to match the actual default behavior, then verify that the documented and observed truncation windows agree.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100